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ABSTRACT 


Two  working  documents  for  Office  of  C..vil  Defense  use  in 
producing  guidance  materials  were  prepared.  They  represent 
proposed  approaches  to  guidance  and  evaluation  documents  for  use 
in  developing  community  shelter  systems. 


i  i 


SUMMARY 


i 


Purpose 


The  primary  purpose  of  this  study  was  to  develop  an  evaluation 
instrument  which  could  be  used  to  assess  the  operational  capabilities  of 
community  fallout  shelter  systems.  A  secondary  purpose  was  to  translate 
the  basic  data  jsed  in  constructing  the  evaluation  instrument  into  guidance 
material  useful  to  local  civil  defense  planners. 

Approach 

Information  on  the  characteristics  of  community  shelter  systems  was 
derived  through  literature  review,  interviews  with  knowledgeable  civil 
defense  officials,  and  the  systematic  consideration  of  the  gross  tasks 
involved  in  system  development,  maintenance,  and  operation.  From  these 
basic  data,  487  characteristics  (hereafter  referred  to  as  plan  factors) 
were  written  up  as  separate  items  in  an  evaluation  instrument.  An  estimate 
of  the  importance  of  each  plan  factor  was  made  by  a  group  of  thirteen  raters. 
These  Judgments  served  as  the  basis  for  developing  item  weights.  The  item 
weights  were  subsequently  incorporated  into  a  scoring  method  for  the 
instrument.  The  evaluation  instrument  was  then  applied  to  two  sets  of 
plans  for  community  shelter  systems. 

The  guidance  material  for  local  civil  defense  planners  was  prepared 
in  workbook  form.  A  brief  discussion  of  each  plan  factor  was  written  for 
each  plan  factor.  Accompanying  each  such  discussion  was  a  "decision" 
statement  paraphrasing  the  essential  aspects  of  the  factor  as  they  would 
apply  to  a  shelter  system.  Also,  space  was  left  for  including  comments 
regarding  specific  shelters  within  a  system. 

Results 


1.  The  evaluation  instrument  which  was  developed  proved 

capable  of  assessing  the  strong  and  weak  points  of  plans 
for  cotanunity  shelter  systems. 


iii 


2.  No  particular  difficulties  were  encountered  in  applying 
the  instrument,  or  scoring  the  results  of  the  evaluations. 

3.  No  system  characteristics  appeared  in  the  planning  docu¬ 
ments  which  were  not  adequately  covered  by  plan  factors  in 
the  instrument. 

4.  The  guidance  material  document  which  was  prepared  presents 
an  integrated  source  of  information  applicable  to  the 
development  of  a  community  shelter  system. 
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A  GENERAL  OVER -VIEW  OF  THE  STUDY 


Background 

Positive  action  toward  a  dynamic  national  shelter  capability  is  now 
being  taken  by  the  Federal  government  in  the  form  of  the  National  Fallout 
Shelter  Marking  and  Stocking  program.  In  addition,  a  few  communities 
throughout  the  country  have  considered  how  a  nuclear  attack  would  affect 
them,  and  to  varying  degrees  they  have  initiated  planning  activities 
oriented  towards  their  own  protective  needs.  It  is  expected  that  as  more 
shelter  space  is  designated  (or  developed)  and  stocked  during  the  course 
of  the  marking  and  stocking  program,  more  communities  will  begin  to  plan 
intensively  and  to  act  to  provide  protection  from  nuclear  attack  for  their 
entire  populations. 

Whether  a  community  elects  to  designate  and  modify  existing  buildings 
as  shelters,  or  to  construct  shelters,  there  must  be  a  we 11 -coordinated  and 
controlled  system  of  sheltere  capable  of  protecting  and  sustaining  the 
entire  population  of  the  community.  For  the  purposes  of  this  study,  the 
objective  of  such  a  system  may  be  stated  as  follows:  "The  purpose  of  a 
community  shelter  program  is  to  protect  the  people  of  that  community  by 
preserving  their  respective  capacities  to  a  degree  which  will  enable  them 
to  survive  an  enemy  attack  and  return  to  near  normal  capacity  for  produc¬ 
tivity  in  the  shortest  possible  time."  The  effectiveness  of  a  community 
shelter  system  in  accomplishing  this  objective  will  depend  to  a  great  extent 
upon  how  well  it  can  function  as  a  system. 

The  term  system,  as  used  in  this  study,  is  analogous  to  its  use  in 
reference  to  a  weapon  system.  Developing,  operating,  and  maintaining  a 
system  involves  very  similar  steps  regardless  of  the  nature  of  that  system. 
For  example: 

1.  System  objectives  must  be  determined. 

2.  Plans  must  be  formulated  for  the  achievement  of  these 
objectives . 

3.  There  must  be  a  developmental  period. 

4.  The  operational  capabilities  of  the  system  must  be  evaluated 
to  Insure  that  it  can,  in  fact,  accomplish  its  objectives. 

5.  Once  a  system  is  operational,  logistic,  maintenance  and  manage¬ 
ment  requirements  must  be  met  throughout  the  life  of  the  system. 


The  system  approach  to  developing  a  community  shelter  program  is  the 
concept  on  which  work  on  this  project  is  based.  All  research  products  cf 
the  project  are  aimed  at  enhancing  the  operational  capability  of  community 
fallout  shelter  systems. 

Purpose 


The  primary  purpose  of  this  study  has  been  to  develop  an  evaluation 
instrument  which  could  be  U3ed  to  assess  the  operational  capability  of 
community  fallout  shelter  systems.  A  secondary  purpose  has  been  to  bring 
together,  under  one  cover,  information  regarding  the  characteristics  of  an 
effective  shelter  system.  This  information  is  presented  in  a  form  which 
local  civil  defense  personnel  may  conveniently  follow  when  developing  their 
plans . 

It  was  originally  planned  that  a  central  purpose  of  this  study  would 
be  to  utilize  the  evaluation  instrument  to  be  developed  to  appraise  several 
representative  community  plans  for  using  community  shelter  facilities 
developed  to  date  under  the  national  shelter  program,  including  the 
Government -sponsored  prototype  shelter  construction  program.  However,  it 
became  apparent  very  early  in  the  contract  period  that  civil  defense-  plan¬ 
ning  and  shelter  construction  at  the  community  level  were  not  sufficiently 
developed  to  permit  full-scale  evaluations.  Where  prototype  shelters  were 
under  construction,  no  comprehensive  plans  for  their  use  or  integration 
into  a  community  shelter  system  were  available.  Conversely,  a  few  commu¬ 
nities  had  accomplished  some  preliminary  planning  toward  a  community  shelter 
system  but  had  no  actual  shelter  facilities. 

In  view  of  the  situation,  a  shift  in  emphasis  and  consequent 
realignment  of  the  contemplated  research  was  suggested  and  subsequently 
approved.  This  revised  plan  called  for  (l)  development  of  a  more  refined 
evaluation  instrument  having  broader  scope  and  longer  usefulness  than 
contemplated  for  the  instrument  conceived  initially,  and  (2)  preparation 
of  an  additional  document  to  consist  of  information  gathered  during  the 
development  of  the  evaluation  instrument  and  to  be  packaged  in  such  a  way 
as  to  have  instructional  value  for  local  civil  defense  planners. 
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Research  Products 


Evaluation  Instrument.  The  completed  Evaluation  Instrument  contained 
U87  separate  plan  factor  items.  Each  plan  factor  represented  a  character¬ 
istic  or  aspect  of  a  community  shelter  system.  Plan  factor  items  were 
organized  into  33  subject-matter  categories  within  the  Evaluation  Instru¬ 
ment.  This  grouping  of  plan  factors  by  subject-matter  categori.es  (and  in 
some  cases  subcategories)  served  to  provide  a  classification  system  whereby 
the  user  could  readily  locate  a  particular  plan  factor.  It  also  provided 
a  basis  for  summarizing  the  results  of  an  evaluation  in  terms  of  major 
system  functions  such  as  Shelter  Management  and  Radiological  Defense. 

In  using  the  Evaluation  Instrument,  the  evaluator  must  determine 
whether  or  not  each  plan  factor  has  been,  or  will  be,  provided  for  in 
planning  and  developing  the  shelter  system.  For  each  plan  factor,  an  all 
or  none  decision  thus  must  be  made.  Although  it  would  be  desirable  to 
obtain  a  more  precise  rating  for  each  plan  factor  in  accordance  with  the 
degree  of  adequacy  reached  in  the  plan  being  evaluated,  little  or  no  objec¬ 
tive  data  are  available  at  this  time  for  use  in  making  such  precise  discrim- 
inationa  However,  when  data  are  available  tc  indicate  the  minimal  degree 
Gf  adequacy  in  planning  required  to  give  credit  for  the  item,  such  data  are 
identified  under  the  plan  factor  in  the  Evaluation  Instrument;  there  data 
are  labelled  "STANDARD"  in  each  instance,  to  show  that  the  data  will  assist 
the  evaluator  in  making  the  required  yes  or  no  decision. 

A  scoring  method  was  developed  for  the  Evaluation  Instrument  which 
would  provide  quantitative  estimates  of  the  operational  capabilities  of  the 
shelter  systems  evaluated.  For  each  plan  factor  item,  the  Evaluation 
Instrument  shows  the  weighted  score  assigned  to  the  item;  this  score  corre¬ 
sponds  with  the  Judged  importance  of  the  item  to  system  effectiveness,  as 
determined  by  a  group  of  Judges. 

The  Evaluation  Instrument  is  intended  as  an  aid  to  Office  of  Civil 
Defense  personnel  and  state  CD  planners  with  the  responsibility  for  assess¬ 
ing  the  operational  capability  of  community  fallout  shelter  systems.  The 
instrument  is  designed  to  be  applicable  in  evaluating  systems  which  vary 
across  a  wide  range  In  parameters,  such  as  community  size  and  shelter  types. 
It  can  also  be  used  to  evaluate  shelter  systems  Ir.  all  stages  of  develop¬ 
ment  from  a  completely  "paper"  system  to  one  which  is  fully  operational. 


Guidance  Material.  In  addition  to  the  Evaluation  Instrument,  A  Work¬ 
book  For  Use  hy  Local  Civil  Defense  Officiala  was  developed.  The  workbook 
utilizes  all  the  basic  plan  factor  information  that  went  into  the  prepara¬ 
tion  of  the  Evaluation  Instrument,  presenting  it  in  an  expanded  form  amenable 
to  its  use  as  guidance  material  in  the  development  of  community  shelter 
systems. 

The  guidance  material,  as  presented,  is  expected  to  make  several 
important  contributions.  First,  it  provides  under  one  cover  a  condensed 
and  readable  listing  of  th"  multitude  of  specific  factors  which  must  be  con¬ 
sidered  when  developing  a  community  shelter  system.  Second,  the  material 
is  presented  in  such  a  way  that  if  directions  for  using  it  are  followed, 
most  of  the  basic  decisions  necessary  in  the  development  of  a  shelter  system 
will  have  been  made.  And  third,  guidance  in  shelter  system  development  is 
provided  in  "extra-shelter"  areas  which  have  received  little  attention  in 
the  past. 

Suggestions  for  Future  Research 

The  Evaluation  Instrument  developed  during  the  course  of  this  study 
is  capable  of  assessing  the  shelter  system  planning  efforts  which  have 
been  carried  out  to  date  and  those  which  can  be  expected  for  some  time  to 
come.  The  guidance  material  prepared  during  the  study  is  directed  at  ful¬ 
filling  a  perceived  need  for  an  integrated  presentation  of  the  basic  date 
needed  to  plan  a  community  shelter  system.  They  represent  initial  efforts 
ir.  both  these  areas. 

As  the  results  of  current  and  future  research  on  specific  aspects  of 
civil  defense  become  available,  they  should  be  incorporated  Into  revisions 
of  the  guidance  material  and  the  evaluation  instrument  to  enhance  their 
usefulness.  Suggestions  regarding  how  this  might  be  accomplished  on  an 
interim  basis  may  be  found  in  Appendix  C. 

Several  additional  areas  of  guidance  and  evaluation  should  also  be 
researched.  Among  these  are  the  specific  efforts  listed  bel^w.  They  are 
discussed  in  greater  detail  in  the  future  esearch  section  of  this  report. 

1.  More  definitive  levels  of  adequacy  should  be  determined 
for  plan  factors. 


2.  The  effects  of  plan  factor  interactions  should  be  deter¬ 
mined  and  a  means  of  evaluating  such  interactions  developed 
and  incorporated  into  the  total  shelter  syetem  evaluation 
scheme. 

3.  An  intensive  3tudy  should  be  conducted  to  identify  the 
critical  aspects  of  "extra-shelter"  functions  of  the 
shelter  systems,  such  as  post -shelter  planning. 

4.  Present  and  future  research  such  as  suggested  in  3  above 
should  be  incorporated  into  a  comprehensive  guidance 
"package"  which  would  outline  in  detail  the  consecutive 
steps  required  to  develop  a  community  shelter  system. 
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EVALUATION  INSTRUMENT 


Development 

This  section  of  the  report  describes,  in  turn,  the  series  of  steps 
taken  in  the  total  process  of  developing  the  evaluation  instrument.  For 
convenience,  each  major  step  in  the  process  is  given  an  indented  sub-heading. 

Identification  and  Organization  of  Plan  Factors.  The  first  step  in 
the  development  of  the  Evaluation  Instrument  was  to  identify  those  factors 
which  are  characteristics  of  an  effective  shelter  system.  During  the 
initial  phases  of  this  task,  three  general  guidelines  were  evolved  to  give 
direction  to  the  effort. 

First,  a  generally  stated  objective  for  community  shelter  systems 
was  prepared.  It  read  as  follows :  "The  purpose  of  a  community  shelter 
program  is  to  protect  the  people  of  that  community  by  preserving  their 
respective  capacities  to  a  degree  which  will  enable  them  to  survive  an 
enemy  attack  and  return  to  near  normal  capacity  for  productivity  in  the 
shortest  possible  time."  From  this  standpoint,  not  only  specific  sheltering 
functions  but  also  most  life -support  functions  a  community  normally  provides 
for  its  people,  excluding  obvious  luxuries,  were  candidates  for  plan  factors. 

Second,  an  outline  of  subject-matter  categories  and  subcategories 
was  prepared  to  s  ructure  the  search  for  relevant  plan  factors  and  to  help 
classify  the  factors  as  they  were  identified.  New  categories  were  added, 
and  rewording  cf  the  original  categories  was  carried  out  as  necessary 
throughout  tna  p-ocess  of  identifying  plan  factors.  This  organizational 
superstructure  for  the  plan  factor  data  also  helped  Insure  a  complete  and 
balanced  coverage  of  all  aspects  of  community  shelter  systems. 

Third,  in  identifying  and  writing  up  the  plan  factors,  each  factor 
wa6  made  as  independent  of  other  factors  as  possible.  When  subject-matter 
overlap  occurred,  appropriate  instructions  for  coping  with  its  effects 
were  included  in  the  Evaluation  Instrument. 

Within  the  framework  of  these  general  guidelines,  plan  factor  data 
were  collected  by  the  following  means: 

1.  Search!  .  the  literature  on  civil  defense  and  related  areas. 

This  method  collecting  plan  factor  data  was  continued 

until  review  of  additional  documents  contributed  no  new  plan 
factors . 
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J i.  Reviewing  available  local  civil  defense  planning  documents. 

3.  Discussing  plans  and  planning  efforts  with  civil  defense 
officials  with  special  emphasis  on  plan  factors.  Regional, 
state,  and  local  civil  defense  officials  in  California  were 
contacted  regarding  any  shelter  system  planning  and  develop¬ 
ment  activities  being  undertaken.  Several  additional  plan¬ 
ning  documents  pertaining  to  community  shelter  systems  were 
obtained  through  these  contacts. 

4.  Systematically  considering  the  various  aspects  of  a  community 
shelter  system  and  anticipating  the  need  for  plan  factors. 

This  approach  to  identifying  plan  factors  involved  the  use 

of  a  very  gross  system  analysis  procedure.  Each  aspect  of 
a  community  shelter  system  was  considered  in  terms  of 
activities  or  gross  tasks  necessary  to  accomplish  it.  In 
this  way  obvious  deficiencies  in  plan  factor  coverage  were 
spotted  and  corrected. 

When  no  new  plan  factors  were  identified  as  additional  documents 
were  searched,  emphasis  was  shifted  to  editing  the  plan  factor  data  which 
had  been  collected.  Each  plan  factor  wis  reviewed  to  correct  for:  (a) 
redundancy  among  items,  (b)  partial  subject-matter  overlap  among  items, 

(c)  errors  in  subject-matter  classification,  (d)  lack  of  clarity  and  con¬ 
ciseness,  (e)  too  coarse  a  categorization,  and  (f)  failure  to  provide  a 
basis  for  evaluation. 

This  editing  of  items  was  completed  before  the  items  were  submitted 
to  judges  for  the  purpose  of  developing  a  criticality  rating  for  each  item. 

Compilation  of  Standards  Information.  Concurrent  with  identifying 
plan  factors,  data  were  collected  which  could  be  used  as  standards  against 
which  the  evaluator  could  judge  the  provisions  made  for  the  plan  factors 
by  the  community  whose  shelter  system  was  being  evaluated. 

It  has  already  beet:  shown  why  ail-or-none  estimates  of  the  provisions 
made  for  plan  factors  are  necessary  during  an  evaluation.  In  addition, 
little  CD  literature,  experimental  or  otherwise,  was  available  to  help  set 
standards  for  giving  all-or-none  credit  for  each  item.  However,  a  tryout 
of  the  evaluation  instrument  subsequently  developed  indicated  that  the 
evaluation  of  provisions  for  plan  factors  on  a  present -absent  basis  was 
feasible  until  additional  research  yields  more  complete  standards  data. 


In  light  of  the  type  of  standards  information  available  and  the 
fact  that  a  dichotomous  (present -absent)  evaluation  of  each  plan  factor  was 
practicable,  standards  for  the  plan  factors  were  handled  as  follows: 

a)  When  the  standard  was  subjective  in  nature,  the  plan  factor 
was  written  to  include  the  standard.  Objective  or  quanti¬ 
tative  standards  data  which  could  be  expressed  in  a  short 
phrase  were  also  incorporated  into  the  appropriate  plan 
factors. 

b)  Where  quantitative  standards  information  was  available  but 
not  amenable  to  inclusion  in  the  plan  factor  statement,  it 
was  presented  below  the  factor  to  which  it  pertained  and 
identified  as  a  standard. 

Format.  Basic  decisions  regarding  the  format  of  the  Evaluation 
Instrument  were  made  early  in  its  development.  When  it  became  evident 
that  approximately  500  separate  plan  factors  would  be  needed  in  a  compre¬ 
hensive  instrument,  a  checklist  type  format  was  deemed  the  most  feasible 
way  of  presenting  the  data.  The  subject-matter  groupings  (categories)  of 
plan  factors  utilized  during  the  process  of  identifying  plan  factors  were 
maintained  as  a  means  of  organizing  the  factors  within  the  Evaluation 
Instrument.  When  appropriate,  subcategories  were  also  identified  to  enhance 
the  usability  of  the  instrument. 

Plan  factor  categories  are  not  specifically  arranged  according  to 
any  definable  phases  of  a  national  emergency  such  as  pre -attack,  attack, 
post -attack,  recovery.  The  primary  reason  for  this  is  that  the  specific 
plan  factors  under  each  category  describe  activities  which  should  be  accom¬ 
plished  ir.  the  development  and  maintenance  of  a  shelter  system  long  before 
an  emergency  exists.  However,  some  factors,  although  having  a  common 
subject-matter  background,  are  pertinent  either  to  "pre -emergency1'  phases 
of  shelter  system  development,  or  to  actual  shelter  operation,  but  not  to 
both.  In.  these  cases,  two  categories  are  used,  and  supplementary  labels 
of  "pre -emergency"  and  "in-shelter"  are  applied.  Examples  of  this  situation 
are  Jhelter  Management  and  Fost-Jhelter  Planning. 

m  complete  list  of  all  categories  used  in  the  Evaluation  Instrument 
is  provided  below.  Definitions  ac  such  are  not  provided  because  the 
categories  are  operationally  defined  by  the  factors  they  contain. 
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GENERAL 

OPERATIONAL  PLAN 

plan/community  COMPATIBILITY 

POPULATION  INDOCTRINATION  AND  TRAINING 

SHELTER  ASSIGNMENTS 

SHELTER  MANAGEMENT  (Pre -Emergency) 

SHELTER  STOCKING 

SHELTER  DESIGN 

SHELTER  UTILIZATION  PLAN 

PERIODIC  MAINTENANCE 

POST -SHELTER  PLANNING  (Pre -Emergency) 

COORDINATION 

WARNING  SYSTEM 

INGRESS 

RADIOLOGICAL  DEFENSE 

SHELTER  MANAGEMENT  ( In-Shelter) 

ATMOSPHERE  CONTROL 


WATER 

FOOD 

SLEEP 

SANITATION 

MEDICAL 

SPACE -VOLUME  REQUIREMENTS 

LIGHTING 

POWER  SUPPLY 

CONTINGENCY  PLANNING 

COMMUNICATION 

CONTROL 

MAINTENANCE 

NOISE 

TRAINING 

RECREATION  AND  RELIGIOUS  ACTIVITIES 
POST -SHELTER  PLANNING  (in-Shelter) 


In  keeping  with  the  checklist  formt.t  of  the  Evaluation  Instrument, 
individual  plan  factor  items  are  presented  in  the  form  of  questions. 

Three  possible  responses  are  provided  for  each  item.  They  are:  "not 
applicable"  if  the  particular  factor  does  not  apply  to  the  system  under 
evaluation;  "yes"  if  the  factor  has  been  provided  for;  and  "no"  if  it  has 

not.  The  symbols  N/A  and  Y  are  used  for  "not  applicable"  and  "yes," 

respectively.  A  number  representing  the  item  weight  is  used  to  indicate 
the  "no"  resp.nse,  The  reasons  for  assigning  the  item  weight  to  a  "no" 

response  rather  chan  to  a  "yes "  response  are  explained  later. 

All  plan  factor  items  except  those  strictly  related  to  planning 
functions  are  stated  in  future  tense.  This  was  dene  to  enhance  the  flex¬ 
ibility  of  the  Evaluation  Instrument,  Ily  a  simple  shift  in  the  verb  tense 
^'f  appropriate  plan  factors,  the  instrument  can  be  applied  to  u  shelter 
system  in  any  stage  of  development  (e.*t. ,  preliminary  planning,  detailed 
>!'  advanced  planning,  developmental,  operational).  Then  too,  most  of  the 
shelter  systems  to  be  evaluate  1  for  some  time  to  'ome  will  be  in  the 
planning  phases  of  development..  Thus,  future  tense  usage  is  most  appropri 
ate  for  the  present  Evaluation  Instrument. 
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Estimation  of  Plan  factor  Criticality.  Several  methods  of  obtaining 
a  quantitative  measure  of  shelter  system  effectiveness  were  considered 
during  the  course  of  the  study.  Basic  to  all  was  the  procedure  of  rating 
each  plan  factor  according  to  the  probable  consequence  or  effect  of  its 
absence  from  the  system. 

One  approach  which  appeared  to  have  considerable  promise  early  in 
the  study  involved  the  use  of  consequence  areas  as  a  means  of  classifying 
and  ultimately  quantifying  the  effect  of  not  providing  for  each  plan  factor. 
Examples  of  consequence  areas  were  death  or  extreme  debilitation,  danger 
to  physical  health,  and  mental  or  emotional  strain. 

A  preliminary  set  of  consequence  areas  was  derived,  and  an  attempt 
was  made  to  classify  the  plan  factors.  Problems  which  defeated  the  use  of 
this  approach  were: 

a)  The  fact  that  the  absence  cf  many  of  the  plan  factors  could 
be  classified  as  having  several  consequences. 

b)  In  attempting  to  refine  the  consequence  areas  to  be  more 
specific,  the  number  of  consequence  areas  approached  the 
number  of  plan  factors. 

When  initial  attempts  at  specific  classification  of  plan  factor 
consequences  failed,  a  more  general  classification  schema  was  derived, 
and  the  approach  ultimately  used  to  obtain  quantitative  measures  was 
developed.  Three  major  objectives  of  the  approach  were: 

1)  To  provide  a  quantitative  basis  for  further  development 
of  a  scoring  method  for  the  Evaluation  Instrument. 

2)  To  determine  whether  plan  factor  criticality  is  affected 
by  the  conditions  under  which  a  shelter  system  operates. 

3)  "3  obtain  information  useful  in  the  further  screening  of 
plan  factors  to  be  Included  in  the  final  form  of  the 
Evaluation  Instrument. 

Briefly,  the  steps  taker,  in  developing  criticality  ratings  for  the 
plan  factors  were  as  follows: 

1.  A  total  of  493  plan  factor  items  were  assembled  in  a  data  collec¬ 
tion  form.  A  sample  page  from  the  form  is  shown  in  Figure  4,  Appendix  A. 

Item  format  and  arrangement  in  the  data  collection  form  were  nearly  Identical 
t:  those  used  in  the  final  version  of  the  Evaluation  Instrument. 
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2.  A  system  objective  was  formulated  and  four  levels  of  criticality 
were  defined.  The  system  objective  served  as  a  criterion  agaiubt  which  the 
level  of  criticality  of  individual  plan  factors  could  be  Judged.  Each  of 
the  four  levels  of  criticality  was  assigned  a  weight  (3,  2,  1,  and  0, 
respectively,  from  most  to  least  critical).  The  system  objective  and 
criticality  levels,  along  with  the  instructions  to  the  raters  regarding 
their  use,  are  shown  in  Figure  1,  Appendix  A. 

3.  Two  "conditions"  were  described  under  which  the  criticality 
ratings  were  to  be  assigned.  Condition  1  described  an  urban  community  with 
a  highly  developed  industrial  complex  located  near  prime  targets.  Condition 
2  described  a  rural  community  a  considerable  distance  from  prime  targets 
and  subject  only  to  fallout.  These  two  conditions  were  meant  to  describe 
points  near  opposite  ends  of  a  continuum  of  possible  condition?  under  which 
shelter  systems  would  have  to  operate.  Detailed  descriptions  of  the  two 
conditions  are  presented  in  Figures  2  and  3  in  Appendix  A. 

U.  A  total  of  thirteen  (13)  Judges  rated  the  criticality  of  the  plan 
factors.  Six  (6)  raters  performed  the  rating  task  under  Condition  1,  and 
seven  (7)  rated  the  factors  under  Condition  2. 

5.  An  analysis  of  the  rating  data  was  conducted  to  determine  if 
significantly  different  levels  of  criticality  were  assigned  to  the  same 
plan  factors  under  the  two  widely  differing  conditions.  A  brief  summary  of 
the  results  of  the  analysis  will  be  presented  at  this  time.  A  more  detailed 
discussion  of  the  analysis  may  be  found  in  Appendix  B. 

As  might  be  expected,  rater  bias  towards  consistently  high  or  low 
ratings  was  encountered.  Tc  correct  for  this,  the  raw  rating  scores  of 
3,  2,  1  and  0  were  converted  to  standard  scoi  ;  for  each  rater.  Mean 
standard  score  ratings  and  standard  deviations  were  calculated  for  individ¬ 
ual  factcr-3  under  conditions  1  and  2.  "Student's"  t  test  comparisons  were 
made  on  each  factor  to  determine  if  the  mean  criticality  ratings  under  the 
two  conditions  were  significantly  different.  None  of  the  factors  were 
rated  significantly  different  at  the  .01  level  under  the  two  conditions. 

As  a  further  check,  the  raw  score  patterns  of  the  ratings  on  each  factor 
were  -in pared  using  Fisher's  i xact  Probability  Test\  Again,  no  aignifi- 
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cant  differences  were  found  at  the  .01  level.  The  exceptions  to  these 
findings  were  seven  items  where  the  instructions  to  the  raters  made  a 
significant  difference  inevitable. 

These  findings  indicated  that  plan  factor  criticality  should  not  be 
affected  by  the  conditions  under  which  shelter  systems  operate.  This  meant 
that  the  Evaluation  Instrument  could  be  applied  to  shelter  eystems  varying 
across  a  wide  range  of  conditions,  using  a  single  set  of  weights  for  the 
plan  factor  items.  The  alternative  would  have  been  a  complicated  set  of 
varying  item  weights  to  match  varying  shelter  system  conditions. 

Since  the  ratings  under  conditions  1  and  2  were  not  significantly 
different,  a  mean  standard  score  rating  was  calculated  for  each  factor, 
using  the  criticality  rating  data  from  all  thirteen  raters.  This  provided 
the  needed  quantitative  data  for  development  of  the  scoring  method  for  the 
Evaluation  Instrument. 

Six  (6)  plan  factor  items  wero  deleted  from  the  final  form  of  the 
Evaluation  Instrument  as  a  result  of  receiving  a  2ero  (c)  rating  by  all 
thirteen  raters.  By  definition,  these  factors  were  contributing  nothing  to 
shelter  system  effectiveness. 

In  retrospect,  it  should  be  noted  that  some  pre-selection  of  plan 
factors  took  place  during  the  plan  factor  identification  phase  of  the  study. 
That  is,  factors  of  a  luxury  nature,  or  those  obviously  not  important,  were 
excluded  from  consideration.  Even  so,  a  normal  distribution  of  plan  factor 
criticality  was  expected  and  obtained.  A  frequency  distribution  of  mean 
standard  score  ratings  for  the  plan  factors,  as  shown  in  Figure  6,  Appendix 
B,  illustrates  this.  Those  items  at  the  iouer  end  of  the  distribution 
were  .retained  in  the  Evaluation  Instrument  because  it  was  felt  that  at  this 
point  in  the  development  of  shelter  system  evaluat.on  techniques,  errors 
of  inclusion  ratner  than  exclusion  were  more  prudent.  To  compensate, 
adequate  allowances  for  differences  in  relative  importance  among  plan  factors 
were  made  in  the  scoring  method. 

Scoring  method.  As  a  prelude  to  the  actual  development  of  a  scoring 
method  for  the  Evaluation  Instrument,  objectives  which  the  oring  method 
should  accomplish  were  outlined.  These  objectives  arc  presented  below: 

1.  Shelter  systems  should  not  be  penalized  when  fa. tors  not 
applicable  to  them  are  omitted, 
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2.  The  scoring  method  should  provide  a  numerical  translation 
of  a  shelter  system's  operational  capability  with  respect 
to  each  category  of  plan  factors. 

3.  Plan  factors  whose  importance  to  the  operational  capability 
of  a  shelter  system  was  judged  to  be  very  high  must  have  a 
weight  which  reflects  this  importance. 

b.  Providing  for  less  important  plan  factors  must  not  be 
allowed  to  equate  for  the  absence  of  a  highly  critical 
factor  if  such  absence  could  lead  to  system  failure. 

5.  Failure  to  provide  for  factors  whose  omission  could  result  in 
system  failure  should  be  penalized  heaviiy  by  the  scoring  method. 
With  these  objectives  in  mind,  the  scoring  method. to  be  discussed  was 
developed. 

To  preclude  penalizing  shelter  systems  for  not  providing  for  plan 
factors  invalidated  by  specific  conditions,  possible  responses  to  each  plan 
factor  item  include  "not  applicable.”  An  N/A  or  "not  applicable"  response 
to  a  given  plan  factor  item  may  be  appropriate  for  several  reasons.  The 
use  of  this  response  choice  is  discussed  in  detail  later  on  in  Appendix  C 
when  utilization  of  the  evaluation  Instrument  is  reviewed. 

Throughout  the  development  of  the  Evaluation  Instrument,  the  basic 
structure  of  the  scoring  method  was  under  consideration.  The  choice  was 
between  (a)  deriving  a  single  overall  score  to  represent  a  shelter  system's 
operational  capability,  or  (b)  developing  a  profile  scoring  method  whereby 
the  level  of  operational  capability  or  effectiveness  could  be  presented 
by  Pl&n  factor  category.  The  second  method  was  chosen  because  it  facilitates 
a  diagnostic  approach  to  shelter  system  evaluation.  Emphasis  on  the  diag¬ 
nostic  approach  to  shelter  system  appraisal  is  a  salient  characteristic  of 
the  Evaluation  Instrument. 

In  the  scoring  method  developed,  a  score  is  computed  for  each  plan 
factor  category.  These  scores  are  then  displayed  graphically  on  a  profile 
chart.  Presentation  of  the  scores  in  this  manner  affords  an  easily  inter¬ 
preted  Illustration  cf  the  strong  and  weak  points  of  a  shelter  system.  An 
example  of  how  the  profile  chart  is  used  is  shown  in  Figure  8  in  Appendix 

■a 

^  * 

Two  measures  were  taken  to  reflect  the  importance  of  highly  critical 
factors,  and  to  prevent  the  provision  for  less  important  factors  from 
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obscuring  the  absence  of  a  highly  critical  factor.  One,  a  curvelinear 
transformation  was  used  in  converting  the  mean  standard  score  criticality 
ratings  of  the  plan  factors  to  item  weights.  And  two,  a  scoring  technique 
was  developed  whereby  the  plan  factor  item  weights  are  subtracted  from  a 
maximum  possible  score  for  the  pertinent  category  when  plan  factors  are 
not  provided  for.  The  combined  result  is  a  scoring  technique  which  allows 
the  drastic  effect  if  failing  to  provide  for  very  important  plan  factors  to 
he  reflected  numerically  without  giving  undue  weight  to  less  important 
factors.  The  shape  of  the  curve  described  by  the  transformation  chosen  to 
convert  ratings  to  item  weights  is  shown  in  Figure  J,  Appendix  B.  A  dis¬ 
cussion  of  the  rationale  behind  the  choice  of  the  transformation  accompanies 
the  figure.  The  conversion  table  which  was  used  to  transform  the  mean 
factor  ratings  to  item  weights  is  shown  in  Table  1,  Appendix  B. 

The  maximum  score  possible  in  each  category  was  arbitrarily  set  at 
100.  Those  factors  assigned  a  criticality  rating  of  3  (absence  could  cause 
system  failure)  by  92$  or  more  of  the  raters  were  given  an  item  weight  of 
100.  Items  given  nearly  this  high  a  composite  rating  received  correspond¬ 
ingly  high  weights  (92,  8l,  70,  etc.)  with  item  weights  decreasing  rapidly 
for  lower  rated  factors  (see  Table  1,  Appendix  B) .  Using  the  scoring  method 
developed,  the  following  situations  could  occur  during  an  evaluation. 

If  a  factor  with  an  item  weight  of  100  is  omitted,  the  highest 
possible  score  for  the  category  containing  that  factor  is  zero  (o).  The 
system  has  failed  regardless  of  how  many  other  factors  in  the  category 
have  been  provided  for.  If  a  category  contains  several  highly  critical 
factors  which  have  not  been  provided  for,  the  category  would  receive  a  high 
negative  score.  Although  a  shelter  system  can  only  fail  once,  so  to  speak, 
negative  category  scores  indicate  a  serious  deficiency  in  the  system  with 
respect  to  these  categories.  The  larger  the  negative  score,  the  more 
serious  the  deficiencies. 

Detailed  instructions  foe  implementing  the  scoring  method  are  pro¬ 
vided  in  the  Instruction  Manual  which  accompanies  the  Evaluation  Instrument. 

Frofile  Chart.  As  previously  stated,  the  evaluation  of  a  community 
shelter  system  .1.3  diagnostic  in  nature.  That  is,  an  evaluation  is  conducted 
to  determine,  first,  if  there  are  deficiencies  in  a  shelter  system,  and 
second,  where  exactly  these  deficiencies  exist.  Although  infomat  on  re¬ 
garding  system  deficiencies  is  available  piecemeal  during  the  c curse  cf  an 


evaluation,  data  of  this  nature  are  most  meaningful  when  displayed  graph¬ 
ically  as  an  integrated  whole.  The  profile  chart  developed  to  comyliment 
the  scoring  method  provides  a  pictorial  overview  of  a  shelter  system's 
operational  capability.  A  sample  of  a  completed  profile  chart  is  shown  in 
Figure  8,  Appendix  C. 

A  bar  graph  format  was  chosen  for  the  profile  chart  because  it 
provides  the  best  contrast  between  .  oil -provided  for  and  deficient  cate¬ 
gories.  The  area  assigned  to  each  category  is  identified  by  the  category 
title  at  the  top  of  the  chart.  Positive  converted  scores  for  each  category 
are  plotted  directly  on  the  chart  using  the  0-100  scale  on  the  side  of  the 
chart.  This  is  not  a  percentage  scale.  Bather,  it  is  a  unit  scale  of 
arbitrary  length  determined  during  the  development  of  the  scoring  method. 
Negative  converted  scores  are  given  a  token  shaded  area  below  the  zero  line 
and  the  actual  negative  score  is  entered  in  the  category's  area  directly 
below  the  shaded  portion.  Token  shaded  areas  and  actual  scores  are  used 
to  indicate  negative  converted  scores  on  the  chart  because  of  the  extreme 
variation  in  possible  negative  scores.  Negative  converted  scores  can  range 
from  z;  re  to  over  700;  this  range  cannot  co^  enient’.y  be  plotted  to  scale 
on  a  small  page. 

Tryout 


At  this  point,  in  the  study  a  complete  but  untried  Evaluation  Instru¬ 
ment  existed.  The  tryout  consisted  of  evaluating  two  shelter  plans  by  use 
of  the  Evaluation  Instrument,  The  following  paragraphs  tell  why  a  tryout 
phase  was  needed  and  how  it  was  carried  out.  Specific  objectives  to  be 
achieved  by  the  tryout  are  listed.  The  results  of  the  tryout  are  discussed 
in  terms  of  the  degree  to  which  each  objective  was  achieved.  Also  included 
is  a  discussion  of  the  deficient  categories  common  to  both  of  the  shelter 
system  plans  used  in  the  tryout. 

Purpose.  The  purpose  of  this  phase  of  the  study  was  to  proof -test 
the  Evaluation  Instrument  as  developed  and  make  any  refinements  necessary. 
Although  tht-  two  available  shelter  system  documents  could  only  bv  classi¬ 
fied  as  representing  preliminary  planning  efforts,  they  were  for  the  most 
part  fairly  detailed  in  nature.  A  satisfactory  verification  of  the  Evalu¬ 
ation  Instrument  w«s  deemed  possible. 


Approach.  The  Evaluation  Instrument  was  applied  to  two  sets  of 
community  shelter  system  planning  documents  which  represented  preliminary 
plann.ng -stage  data.  The  techniques  used  were  exactly  as  described  in  the 
Instruction  Manual  which  accompanies  the  Evaluation  Instrument. 

Specific  objectives  of  the  tryout  were  to: 

1.  Evaluate  plan  factor  items  for  clarity  and  conciseness. 

2.  Test  the  facility  with  which  individual  items  could  be 
applied  to  actual  plans. 

3.  Determine  the  feasibility  of  the  Evaluation  Instrument 
format. 

4.  Check  the  adequacy  of  coverage  of  (a)  plan  factors  within 
the  categories,  and  (b)  the  categories  themselves. 

5.  Evaluate  the  scoring  method. 

6.  Evaluate  the  usefulness  of  the  instrument  in  assessing 
the  operational  capabilities  of  shelter  systems  as  they 
are  described  in  written  form. 

7.  Determine,  if  possible,  whether  written  system  documen¬ 
tation  is  an  adequate  basis  for  an  evaluation  or  whether 
site  visics  may  also  be  required. 

Successful  achievement  of  the  above  ojbectives  would  produce  a 
complete  and  verified  Evaluation  Instrument. 

Results.  Individual  plan  factor  items  were  evaluated  for  clarity 
and  conciseness  by  the  thirteen  raters  during  the  criticality  rating  phase 
of  developing  the  Evaluation  Instrument.  Ambiguous  or  vague  statements 
brought  to  light  at  that  time  were  re-written.  A  second  evaluation  of 
clarity  and  conciseness  during  the  tryout  served  to  double-check  these 
aspects  of  the  plan  factor  items. 

Statements  regarding  previsions  for  individual  plan  factors  in  the 
system  documentation  were  readily  identified  with  the  corresponding  plan 
factor  items  in  the  Evaluation  Instrument.  The  only  problems  encountered 
were  in  locating  the  appropriate  information  in  the  planning  documents. 
Further  discussion  of  this  situation  is  pertinent  to  instrument  format  which 
is  covered  next. 

Organization  of  the  plan  factor  items  into  subject-matter  categories 
did  much  to  alleviate  the  problem  of  matching  plan  factor  items  to  state¬ 
ments  regarding  provisions  for  them  in  the  system  documentation .  At  the 


present  time,  the  classification  of  information  and  its  organization  in 
documents  describing  a  shelter  system  ere  determined  by  the  local  shelter 
system  planners.  As  long  as  this  situation  continues,  the  best  organiza¬ 
tional  format  for  an  evaluation  instrument  is  one  which  facilitates  the 
matching  of  plan  factor  items  and  statements  regarding  provisions  for  them. 
The  subject-matter  grouping  of  plan  factor  items  used  in  the  Evaluation 
Instrument  doe3  facilitate  this  matching  process.  In  the  long  run,  com¬ 
munities  should  be  provided  an  outline  of  the  Evaluation  Instrument  so  that 
planning  documents  would  be  more  standard  in  organization. 

No  individual  plan  factors  or  categories  of  factors  were  found  in 
the  two  shelter  system  planning  documents  evaluated  which  were  not  also 
included  in  the  Evaluation  Instrument.  This  is  not  to  say  that  the  list 
of  plan  factors  in  the  instrument  is  so  extensive  that  new  factors  may  never 
be  identified.  However,  until  communities  have  considerable  more  guidance 
in  the  development  of  a  shelter  system  than  they  do  at  the  present  time, 
the  coverage  provided  by  the  Evaluation  Instrument  will  be  quite  adequate. 

As  more  comprehensive  guidance  becomes  available  to  local  shelter  system 
planners,  the  Evaluation  Instrument  should  be  revised  to  accurately  assess 
the  more  sophisticated  planning  efforts  which  can  be  expected. 

The  method  of  responding  to  individual  items  provided  a  satisfactory 
means  for  recording  the  results  of  the  item-by-item  evaluation.  Category 
total  scores  were  easily  determined.  The  arithmetic  of  summing  item  weights 
to  obtain  category  totals  and  the  calculation  of  converted  scores  for  each 
category  were  easily  accomplished  within  the  formats  provided. 

Evaluation  of  the  two  sets  of  shelter  system  planning  documents 
verified  the  instrument's  ability  to  assess  preliminary  system  planning 
efforts  and  to  clearly  differentiate  levels  of  comprehensiveness  in  such 
planning.  The  sample  Operational  Capability  Profile  depicted  in  Figure  8, 
Appendix  C,  was  prepared  using  the  actual  results  of.  the  evaluation  of  the 
more  detailed  of  the  two  system  planning  documents,  hs  can  be  seen,  great 
’•sriations  in  providing  for  factors  in  the  various  categories  are  reflected 
by  the  profile  chart.  This  checks  well  with  the  author's  impression  of 
these  variations  as  he  conducted  the  tryout.  Although  this  particular 
sheiter  system  did  n  t  make  a  very  good  showing  in  the  evaluation  even 
though  it  was  the  better  of  the  two,  it  must  be  remembered  that  neither  of 
the  planning  documents  was  prepared  as  a  final  operational  plan.  They  did 
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serve  their  purpose,  however,  in  that  they  were  complete  enough  to  allow 
an  evaluation  to  be  accomplished,  showing  the  Evaluation  Instrument  to  be 
effective  in  assessing  preliminary  planning  documents.  System  documentation 
actually  submitted  to  CCD  for  evaluation  will  most  probably  be  much  more 
complete  even  in  the  preliminary  planning  stages. 

Due  to  the  type  of  system  documentation  available  for  use  during 
the  tryout,  an  unqualified  recommendation  regarding  the  sufficiency  of  a 
completely  "paper"  evaluation  of  a  shelter  system,  as  opposed  to  on-site 
visits,  cannot  be  made  at  this  time.  A  great  deal  will  depend  upon  how 
well  all  aspects  of  the  community  shelter  system  to  be  evaluated  are  docu¬ 
mented.  Intuitively,  it  would  seem  that  if  a  shelter  system  were  completely 
documented,  a  satisfactory  "paper"  evaluation  of  that  system  should  be 
possible.  However,  this  assumption  should  be  verified  as  soon  as  system 
documentation  and  corresponding  equipment  and  facilities  are  available  in 
an  operational  community  shelter  system. 

It  is  also  interesting  to  note  the  results  of  the  tryout  in  terms 
of  the  deficient  categories  (those  with  negative  scores)  common  to  both 
community  shelter  systems.  In  general,  they  represent  those  areas  of  civil 
defense  research  and  guidance  which  have  received  the  least  attention  to 
date.  The  categories  are  listed  below  with  comments,  where  appropriate. 

Shelter  Management  (Pre -Emergency)  -  particularly  deficient  with 
respect  to  personnel  selection,  proof -testing,  specific  assignment  to 
shelters,  documentation  of  management  guides,  and  training. 

Shelter  stocking  -  Very  little,  if  anything,  was  mentioned  regarding 
this.  Apparently  communities  are  content  to  leave  this  in  the  hands  of  the 
Federal  government  under  the  marking  end  stocking  program. 

P  riodlc  Maintenance  -  almost  completely  ignored. 

Post -Shelter  Planning  (both  Pre-Emergency  and  In-cbelter)  -  These 
two  categories  were  completely  ignored  with  the  exception  of  o  few  comments 
regarding  immediate  area  decontamination  in  one  of  the  documents.  It 
should  be  re-emphasized,  however,  that  the  documents  evaluated  represented 
such  an  early  stage  in  shelter  system  development  that  the  authors  may  well 
have  felt  this  was  looking  too  far  ahead. 


Warning  System  -  Neither  plan  mentioned  a  civil  defense  warning 
system  although  both  communities  probably  have  them.  Perhaps  communities 
consider  their  warning  systems  as  independent  of  the  shelter  system. 

Rediological  Defense  -  Both  systems  were  weak  in  this  category 
although  each  had  provided  for  factors  the  other  had  not. 

Sanitation  -  For  the  most  part,  the  deficiencies  were  caused  by  not 
providing  for  several  moderately  critical  factors. 

Contingency  Planning  -  As  with  Sanitation,  system  failure  would  be 
attributed  to  failure  to  provide  for  moderately  critical  factors. 

Conclusions 


The  Evaluation  Instrument  developed  under  this  contract  represents 
an  initial  step  in  the  evolution  of  a  technique  for  assessing  shelter 
systems.  The  study  has  resulted  in  a  first  approximation  to  a  comprehensive, 
objective,  and  detailed  procedure  for  making  a  quantitative  evaluation  of 
all  phases  of  the  development,  maintenance,  and  operation  of  community 
fallout  shelter  systems. 

In  its  current  stage  of  development,  the  instrument  represents 
composite  judgments  concerning  the  essential  characteristics  of  shelter 
systems,  the  relative  importance  of  these  characteristics  to  operational 
capability,  and  the  applicability  of  the  instrument  as  it  is.  Civil 
defense,  however,  is  a  dynamic  undertaking.  In  the  present  instrument, 
plan  factor  coverage,  and  evaluative  techniques  are  as  comprehensive  and 
objective  as  funds  and  time  would  permit.  Unfortunately,  in  documenting 
anything  as  dynamic  as  the  subject-matter  at  hand,  written  material  is 
often  in  error  or  otherwise  deficient  before  it  cen  be  published.  For 
these  reasons,  the  Evaluation  Instrument  should  be  continually  updated  to 
reflect  the  latest  results  of  on-going  research. 

Since  this  has  been  a  developmental  effort,  recommendations  in  the 
usual  sense  of  the  word  are  not  in  order.  However,  several  suggestions 
with  respect  t  refining  the  present  instrument  are  appropriate.  These 
may  bt!  found  in  the  section  of  this  report  dealing  with  suggestions  for 
future  research. 
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GUIDANCE  MATERIAL 


Development 

Rationale.  Adaptation  of  basic  plan  factor  information  to  a  guidance 
material  format  was  proposed  for  several  reasons.  First,  there  was  nothing 
resembling  the  proposed  effort  available  in  the  literature  reviewed.  That 
is,  nowhere  did  there  exist  a  comprehensive  yet  condensed  and  readable 
listing  of  the  multitude  of  specific  factors  which  must  be  considered  when 
developing  a  community  fallout  shelter  system. 

Second,  it  was  felt  that  merely  listing  the  characteristics  of  an 
effective  shelter  system  was  not  enough.  Plan  factor  information  should 
be  presented  on  a  concrete  level  so  that  decisions  could  easily  be  made 
regarding  whether  or  not  factors  would  be  provided  for.  Further,  the 
format  for  presenting  guidance  material  should  be  such  that  these  decisions 
had  to  be  made . 

Third,  the  amount,  type,  and  level  of  community  shelter  system 
development  efforts  being  carried  out  in  the  nation  at  the  time  this  study 
began  indicated  a  real  need  for  system-oriented  guidance  of  the  type  pro¬ 
posed.  And  fourth,  community  shelter  systems  patterned  after  the  proposed 
guidance  material  should  provide  adequate  fallout  protection,  and  should  be 
amenable  to  evaluation  by  the  instrument  under  development. 

Source  of  Material.  The  Evaluation  Instrument  and  the  guidance 
material  were  developed  on  the  basis  of  the  same  data.  During  that  phase 
of  the  study  in  which  plan  factors  were  being  identified  in  the  civil 
defers,  and  related  literature,  information  regarding  the  factors  was 
abstracted  in  considerably  greater  detail  than  was  necessary  for  use  in 
the  Evaluation  Instrument  alone.  Considerable  editing  was  required  to 
convert  these  basic  data  to  plan  factor  items  fir  the  Evaluation  Instrument. 
Howevei;  much  material  not  needed  for  the  Evaluation  Instrument  was  useful 
in  preparing  the  guidance  document. 

In  treating  plan  factors  in  the  guidance  material,  details  and 
qualifications  vere  woven  into  the  discussion  of  the  basic  factors  in  such 
a  way  as  to  lend  continuity  to  the  presentation  of  the  material,  and  to 
make  it  specific  enough  so  that  individual  local  civil  defense  planners 
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could  relate  it  to  their  own  situations.  The  common  source  of  background 
material  made  it  possible  to  develop  two  research  products  with  considerably 
greater  scope  than  would  otherwise  have  been  possible. 

Format.  Development  of  a  format  for  the  guidance  material  was 
based  on  the  following  objectives. 

1.  Discussion  of  the  plan  factors  had  to  be  readable, 
reasonabJy  brief,  and  meaningful  to  local  civil  defense 
planners  coping  with  widely  varying  system  conditions. 

2.  The  guidance  material 1  had  to  be  presented  in  such  a 
manner  that  basic  decisions  regarding  a  shelter  system 
for  the  community  would  be  made  automatically  if  the 
instructions  for  its  use  were  followed. 

3.  The  format  must  provide  for  exceptions  to  community- 
level  decisions  with  respect  to  individual  shelters. 

To  accomplish  these  objectives,  A  Workbook  For  Use  by  Local  Civil 
Defense  Officials  was  developed.  The  guidance  material  is  presented  in 
a  columnar  format  in  the  workbook.  In  the  firBt  column  from  the  left,  a 
brief  paragraph  is  devoted  to  the  discussion  of  each  plan  factor  in  turn. 
Organization  of  this  material  within  the  workbook  in  general  parallels 
that  of  the  Evaluation  Instrument,  except  where  some  deviation  is  needed 
to  achieve  continuity  of  discussion. 

Accompanying  the  discussion  of  each  plan  factor  and  directly  to  its 
right  in  the  second  column  of  the  format  is  a  decision  statement  which 
briefly  summarizes  the  discussion  and  presents  it  in  the  form  of  a  decision. 
This  column  is  entitled  "Decisions  for  Community  Shelter  System."  For 
example,  a  plan  factor  concerning  the  cross-training  of  shelter  management 
personnel  is  accompanied  by  the  following  decision  statement*  "Members 
of  each  shelter -management  staff  will  be  cross-trained  so  that  they  can  sub¬ 
stitute,  if  necessary,  for  people  in  other  departments."  The  decision 
statement  is  accepted  or  rejected  for  the  shelter  system.  When  appropriate, 
decision  statements  contain  blanks  to  be  filled  in  by  the  planners.  An 
example  of  this  would  be  a  decision  statement  such  as  the  following: 

"Shelter  will  be  provided  for  (x)  days." 

In  order  to  provide  for  exceptions  to  community -level  decisions  with 
respect  to  individual  shelters,  a  third  column  is  provided.  It  is  labelled 
Modifications  for  Shelter  No.  _ . "  Exceptions  to  or  details  of  the 
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community -level  decision  as  it  relates  to  a  specific  shelter  are  entered  in 
this  column.  Blank  space  is  provided  in  the  column  for  recording  exceptions. 
When  details  specific  to  each  shelter  are  required,  a  decision  statement 
with  appropriate  blanks  is  placed  in  the  third  column.  (Example:  Shelter 
No.  _  will  provide  shelter  for  (X)  days.) 


Use 


The  guidance  material  workbook  was  designed  specifically  for  use  by 
local  civil  defense  planners.  It  is  being  submitted  to  the  Office  of  Civil 
Defense  in  manuscript  form.  The  workbook  represents  a  proposed  approach  to 
a  perceived  need  for  guidance  in  community  shelter  system  development  in  an 
integrated  form  under  one  cover.  If  the  approach  is  acceptable,  it  is  ex¬ 
pected  that  OCD  will  adapt  the  material  to  their  publication  specifications 
and  make  copies  available  to  local  civil  defense  officials. 

Detailed  instructions  for  applying  the  guidance  material  in  develop¬ 
ing  a  community  shelter  system  are  provided  in  the  workbook.  In  general,  the 
guidance  material  is  used  in  two  stages.  First,  decisions  are  made  on  the 
plan  factors  as  they  pertain  to  the  shelter  system  as  a  whole.  Then  after 
appropriate  action  has  been  taken,  each  decision  is  reviewed  in  light  of 
individual  shelter  requirements.  This  procedure  has  the  advantage  of  develop 
ing  a  shelter  system  tailor-made  to  the  facilities  available  or  anticipated. 
It  also  serves  to  double -check  the  practicality  of  system-level  decisions. 

If  too  many  individual  shelters  must  make  exceptions  to  a  system  decision,  it 
may  indicate  the  initial  decision  was  erroneous. 

When  community  planners  have  read  and  responded  appropriately  in 
completing  the  workbook,  most  of  the  decisions  basic  to  the  development  of  a 
community  fallout  shelter  system  will  have  been  made.  This  represents  the 
f  irst  step  in  the  development  of  an  effective  snelter  system.  Subsequent 
steps  will  be  concer-ed  with  implementing  these  decisions. 


SUGGESTIONS  FOR  FUTURE  RESEARCH 


The  suggestions  for  future  research  to  be  offered  here  are  categor¬ 
ized  as  (a)  those  pertinent  to  both  guidance  and  evaluation,  (b)  those 
specific  to  guidance  material,  and  (c)  those  specific  to  the  Evaluation 
Instrument.  They  are  discussed  in  this  order  below. 

Increasing  Coverage  of  Critical  Aspects  of  Shelter  Systems 

Since  the  scope  of  the  present  project  is  broader  than  that  typical 
of  most  shelter  research  projects,  it  is  not  surprising  that  available 
literature  failed  to  provide  the  assistance  needed  in  dealing  with  all 
critical  aspects  of  developing,  maintaining,  and  operating  a  community 
shelter  system.  Research  efforts  to  date  have  for  the  most  part  been  con¬ 
cerned  with  those  aspects  pertaining  to  a  single  shelter  during  the  time 
it  is  in  use  as  a  shelter.  Such  areas  as  periodic  maintenance,  planning 
for  post-shelter  living,  integrated  control  of  a  system  of  shelters,  and 
coordination  with  higher  eschelon  CD  organizations  have  received  less 
research  attention.  As  lve  basic  orientation  of  this  study  was  "evaluation 
in  the  light  of  current  and  developing  knowledge  in  civil  defense,"  it  was 
possible  to  identify  plan  factors  in  the  above-mentioned  areas  only  to  tne 
extent  that  usable  information  on  the  subjects  was  available. 

i.'hile  there  were  some  data  available  on  the  critical  aspects  of 
what  might  be  termed  the  "extra -shelter"  functions,  coverage  was  nut  con¬ 
sidered  to  be  entirely  adequate.  Similarly,  there  was  very  little  such  infor¬ 
mation  available  to  communities  as  guidance  material.  This  was  apparent 
in  the  results  of  the  tryout  of  the  Evaluation  Instrument.  Additional 
research  in  identifying  the  critical  aspects  of  such  areas  as  post-shelter 
planning,  periodic  maintenance,  coordination,  control,  etc.,  is  needed  to 
enhance  both  guidance  and  evaluation. 

Integrating  Guidance  Material 

During  the  course  of  the  present  study,  It  became  apparent  that  an 
integrated  presentation  of  the  basic  data  necessary  to  plan  a  community 
shelter  system  was  needed.  Although  there  is  a  considerable  amount  of 
data  available  at  present  which  is  of  value  to  persona  trying  to  develop 
a  community  shelter  system,  this  inf ormat ior.  is  scattered  throughout  a 


great  number  of  documents  which  often  disagree  on  many  points.  The  manu¬ 
script,  A  Workbook  For  Use  by  Local  Civil  Defense  Officials,  prepared 
during  the  present  study,  represents  a  proposed  approach  for  bringing  this 
information  together  in  an  integrated  form  under  one  cover. 

The  guidance  document  prepared  during  the  present  study  represents 
an  Initial  step  in  what  could  be  the  development  of  an  Integrated  guide  to 
community  shelter  system  development.  Such  a  guidance  "package"  would 
present  a  detailed,  step-by-step  set  of  directions  for  developing,  maintain¬ 
ing,  and  operating  a  community  shelter  system.  Achieving  an  integrated 
guidance  "package"  of  the  magnitude  ar.d  comprehensiveness  suggested  will 
entail;  (a)  refining  present  data,  (b)  increasing  the  coverage  of  critical 
aspects,  (c)  research  on  the  optimum  organization  of  the  guidance  material, 
and  (d)  research  on  the  format  for  presenting  the  data. 

Enhancing  the  Evaluation  Instrument 

Although  work  on  the  present  project  to  date  did  not  reveal  any 
gaps  in  coverage  of  important  factors  as  presented  in  the  evaluation 
instrument  developed,  continued  research  may  provide  information  for 
adding  to  the  plan  factors  or  for  revising  their  weighted  values.  Other 
improvements  in  the  instrument  may  be  made  possible  by  future  research. 

These  improvements  could  include  the  following: 

1)  determination  of  a  continuum  or  at  least  multiple 
levels -of -adequacy  for  each  plan  factor. 

2)  development  of  a  means  of  evaluating  the  interaction 
among  plan  factors. 

3)  refinement  of  a  scoring  method  incorporating  the  results 
of  (a)  and  (b)  above. 

4)  continuous  review  of  the  accuracy  of  the  plan  factor  data. 

The  aforementioned  efforts  are  discussed  in  more  detail  in  the  following 
paragraphs . 

Determining  level- of  *  adequacy  standards  for  the  individual  plan 
factors  will  be  a  tvo-’tage  operation.  First,  the  multitude  of  specific 
findings  from  current  and  anticipated  shelter  research  must  be  reviewed 
and  translated  into  a  form  which  will  facilitate  their  use  in  determining 
levels  of  adequacy.  This  will  be  necessary  primarily  because  neither  the 
research  efforts  nor  reports  of  the  findings  are  oriented  towards  such  a 
specific  use. 


As  the  research  data  are  translated  into  standards,  the  dichotomous 
(present-absent)  approach  to  evaluating  plan  factor  provisions  should  be 
replaced  with  a  system  which  permits  the  quantification  of  each  level  of 
adequacy  at  which  individual  plan  factors  can  be  provided  for.  When 
possible,  several  points  on  a  continuum  of  possible  levels  should  be 
identified  for  the  individual  plan  fac'or.  This  should  be  done  to  allow 
for  flexibility  in  determining  the  minimum  level  of  adequacy,  since  the 
minimum  will,  to  some  extent,  be  dependent  on  community  conditions.  As  a 
supplement,  a  set  of  criteria  should  be  developed  which  will  help  evaluators 
determine  the  minimum  levels  of  adequacy  for  each  shelter  system. 

It  must  be  remembered,  in  conjunction  with  determining  levels  of 
adequacy,  that  not  all  plan  factor  variables  are  continuous.  Many  are 
dichotomous  or  at  most  trichotomous .  These  factors  probably  would  be 
retained  in  their  present  form. 

A  quantitative  investigation  of  ihe  interaction  among  plan  factors 
is  also  needed.  Interaction  between  pai-v  or  small  groups  of  plan  factors 
is  subjectively  apparent  when  reading  th<„.  ?  xst  of  plan  factors.  However, 
a  quantitative  estimate  of  the  effects  of  plan  factor  interaction  is  needed 
to  realize  the  full  potential  of  an  evaluation  instrument.  Of  particular 
concern,  is  the  effect  of  interaction  on  the  importance  attached  to  each 
plan  factor.  This  will  entail  developing  a  means  of  evaluating  plan  factor 
interactions  and  then  incorporating  the  results  into  the  total  evaluation 
schema.  Other  types  of  interaction  which  also  must  be  evaluated  include 
interaction  among  plan  factors,  community  characteristics,  and  the  basic 
assumptions  upon  which  development  of  the  shelter  system  is  based. 

As  information  becomes  available  regarding  levels  of  adequacy  and 
plan  factor  interaction,  the  scoring  method  for  the  evaluation  instrument 
will  have  to  be  modified  considerably.  Integrating  level -of -adequacy  and 
interaction  data  on  the  plan  factors  into  an  overall  evaluation  schema  may 
result  in  a  major  modification  to  item  format  and  possibly  to  the  organization 
of  the  evaluation  instrument. 

A  satisfactory  marriage  of  adequacy  and  Intel  action  data  or.  a  purely 
objective  basis  will  probably  not  be  possible  for  some  time  to  come. 

Init'.al  scoring  techniques  will  most  likely  rely  on  judgments  in  much  the 
same  way  that  the  present  scoring  method  does. 
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Keeping  plan  factor  data  current  will  be  of  primary  concern  regard¬ 
less  of  other  research  contemplated.  The  results  of  current  and  future 
civil  defense  research  should,  of  course,  be  incorporated  into  the  evalu¬ 
ation  instrument  as  they  become  available.  This  should  proceed  concurrently 
with  similar  efforts  on  the  guidance  material.  More  than  a  simple  updating 
process  is  involved,  however.  Although  some  new  factors  will  be  identified, 
most  new  data  will  apply  to  levels  of  adequacy  in  providing  for  plan  factors. 
When  modifying  old  data  or  including  new  information,  the  following  should 
be  considered.  Minimum  requirements  or  standards  should  be  stated  as  such 
only  when  they  have  been  substantiated  by  research  and  will  stand  without 
extensive  revision  for  some  time  to  come.  If  the  data  nave  not  been  sub¬ 
stantiated  and  are  subject  to  change,  they  should  be  identified  as  such. 

This  is  an  important  consideration  for  both  evaluation  and  guidance  material. 
The  credibility  of  minimum  requirements  and  standards  information  is  right¬ 
fully  suspect  when  they  are  subject  to  continual  change  without  notice. 


APPENDIX  A:  DATA  COLLECTION 


INTRODUCTION 


In  this  portion  of  the  research  report,  sample  pages  of  the  instru¬ 
ment  used  to  collect  criticality  rating  data  are  presented. 

The  first  illustration  presents  instructions  that  preceded  both 
versions  of  the  plan  factor  criticality  data  collection  form.  Following 
this,  are  the  two  descriptions  of  the  conditions  under  which  the  criticality 
of  the  plan  factors  was  to  be  rated.  The  last  illustration  in  this  section 
represents  a  sample  page  of  plan  factors  from  the  data  collection  form. 
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CRITICALITY  RATING  FORM 


As  part  of  Project  C-98's  effort  to  develop  an  instrument  which 
may  be  used  to  evaluate  the  effectiveness  of  community  shelter  systems, 
the  plan  factors  which  appear  on  the  following  pages  have  been  collected 
from  many  sources.  Some  of  the  factors  are  obviously  crucial  to  a 
system’s  meeting  its  objectives;  other  factors  are  less  important.  In 
order  to  obtain  quantitative  estimates  of  the  criticality  of  the  factors, 
we  are  asking  a  number  of  persons  to  rate  the  factors  with  respect  to 
the  consequence  of  failing  to  provide  for  them  in  planning  the  shelter 
system.  On  the  following  pages,  spaces  are  provided  for  you  to  write 
a  number  (3,  2,  1,  or  C)  beside  each  factor,  which  reflects  your  estimate 
of  the  factor's  importance  in  meeting  the  objective  of  the  shelter 
system. 


A  short  statement  of  the  system  objective,  and  brief  descriptions 
of  the  "criticality  levels"  associated  with  each  of  the  four  points  on 
the  scale  are  given  here: 


SYSTEM  OBJECTIVE 

Survival  of  the  greenest  percentage  of  the  population  possible, 
with  the  survivors  i.i  condition  to  commence  recovery  operations 
immediately  upon  leaving  the  shelters. 

LEVELS  OF  CRITICALITY 


Rating _ _ Consequence  _ 

3  Failure  to  provide  adequately  for  this  factor  could 
result  in  the  shelter  system's  failing  to  accomplish 
its  objective. 

2  Failure  to  provide  adequately  for  this  factor  could 
seriously  df grade  the  effectiveness  of  the  shelter 
system,  but  would  not  result  in  total  system  failure. 

1  Failure  to  provide  adequately  for  this  factor  could 
moderately  degrade  the  shelter  system,  but  would  not 
serious!}  impair  accomplishment  of  the  system  objec¬ 
tive. 

0  Failure  to  provide  adequately  for  this  factor  would 
probably  he  of  little  or  no  consequence  for  the  sys¬ 
tem's  achieving  its  objective. 


Figure  1.  Instruction  Cheet  frera  Plan  Factor  Criticality  Data 
Collection  Firm 


CONDITIONS  OF  THE  SHELTER  SYSTEM  (l)  ! 

i 

♦ 

The  importance  of  many  of  the  plan  factors  will  vary  depending  i 
upon  such  circumstances  as  population  density  and  proximity  tc  tar¬ 
gets.  To  minimize  this  problem  of  "contingent  criticality,"  certain  ; 
conditions  of  the  shelter  system  are  described  below.  Read  these  | 

circumstances  carefully  and  keep  them  in  mind  as  you  rate  the  impor-  • 

tance  of  the  plan  factors. 

1.  The  shelter  system  to  be  evaluated  is  that  of  a  large 
city  with  a  highly  concentrated  population  and  an 
accompanying  industrial  complex. 

2.  All  types  of  group  shelters  (50  or  more  occupants)  which  j 

are  supported  with  public  funds  are  included  in  the  city's  j 

shelter  system,  i.e.,  single -purpose,  dual-purpose,  and  i 

multiple -purpose .  The  dual-  and  multiple -purpose  shelters  j 
include  some  in  which  the  shelter  function  is  primary  and 
some  in  which  it  is  not  primary.  Small  group  (fewer  than 

50  persons)  and  family  shelters  are  not  included. 

3.  Local  civil  defense  planners  do  not  consider  the  city  itself  ; 

a  prime  target,  but  it  is  bracketed  by  prime  targets  within  ; 
a  50-mile  radius  so  that  blast,  heat,  and  heavy  concentra-  j 
tions  of  radiation  from  fallout  can  be  expected  regardless  , 
of  weather  and  wind  conditions.  ! 

i 

4.  A  warning  time  of  15  minutes  between  signal  and  blast  is  ; 

assumed.  , 

5.  Size  of  weapon  and  type  of  burst  are  not  specified,  but  j 

blast  and  fire  damage  is  expected  to  range  from  moderately 
heavy  to  very  light  across  the  city  depending  upon  distance  ■ 
from  ground  zero  and  upon  the  type  of  structure.  Radiation  j 
levels  will  vary  but  all  may  be  considered  high  enough  so 
that  unprotected  persons  would  die  of  radiation  sickness  in 

a  very  short  time. 

6.  Radiation  levels  will  be  such  that  no  one  will  be  able  to 

leave  the  shelter  for  the  first  four  days  following  the 
attack.  Limited  recovery  operations  can  be  initiated  14  : 

days  after  the  attack. 


Figure  2.  Description  of  Condition  1  from  Plan  Factor  Criticality 
Data  Collection  Form 


-30- 


CONDITIONS  OF  THE  SHELTER  SYSTEM  (2) 


i 

I 

* 

The  importance  of  many  of  the  plan  factors  will  vary  depending  I 

upon  such  circumstances  as  population  density  and  proximity  to  targets.  | 
To  minimize  this  problem  of  "contingent  criticality,"  certain  con-  ! 

ditions  of  the  shelter  system  are  described  below.  Read  these  circum-  i 
stances  carefully  and  keep  them  in  mind  as  you  rate  the  importance  of  I 
the  plan  factors. 

1.  The  shelter  system  to  be  evaluated  is  that  of  a  small  town 
(population  below  5 ,000)  with  low  population  concentration 

and  little  or  no  industrial  complex.  j 

l 

2.  There  are  several  group  shelters  (50  or  more  occupants)  in 

the  system.  Several  of  the  possible  types  of  shelters  are 
represented.  Possible  types  are:  single -purpose,  dual-  j 

purpose  (with  shelter  functions  either  primary  or  secondary),  j 
and  multiple -purpose  (with  shelter  function  primary  or  not  i 
primary) .  These  conditions  are  prescribed  to  preclude  j 

simple  solutions  such  as  one -shelter  communities,  or  a  homo-  ! 
geneous  shelter  system.  j 

I 

3-  The  community  is  definitely  not  a  prime  target  and  is  more 

than  100  miles  distant  from  anything  that  might  be  considered  j 
a  prime  target.  However,  prime  targets  are  located  with  j 

relation  to  the  community  so  that  radiation  levels  from  fall¬ 
out  will  be  such  that  radiation  protection  must  be  provided 
for  the  people  of  the  community.  There  are  no  specific  re-  | 
quirements  for  blast  or  heat  protection.  j 

U.  A  minimum  of  h5  minutes  is  assumed  to  be  available  between 

the  time  the  warning  that  the  nation  is  under  attack  is  re-  , 
ceived  and  the  first  traces  r f  fallout  begin  to  arrive.  I 

5.  Radiation  levels  will  be  such  that  emergency  operations  may  j 
be  conducted  outside  the  shelter  two  days  after  the  attack  \ 
and  full  scale  recovery  operations  can  commence  14  days 
after  the  attack.  i 


Figure  3. 


Description  of  Condition  2  from  flan  Factor  Criticality 
Data  Collection  Form 


GENERAL  (Ol) 


1.  If  permanent  and  completely  adequate  shelter 
facilities  are  not  yet  available  for  everyone 
in  the  community,  have  steps  been  taken  to 
provide  interim  shelter  facilities? 


itiJ  i 


i. 


3 


2.  Has  an  analysis  of  such  variables  as  the 
proximity  of  targets,  the  probable  priority 
of  such  targets,  and  the  prevailing  winds 
been  conducted  to  determine  what  effects  of 
a  nuclear  attack  are  possible? 


2. 


I 

3L  i 


3.  Have  specific  assumptions  been  mace  regard¬ 
ing  the  type  of  effects  (blast,  heat,  radia¬ 
tion)  for  which  protection  is  to  be  afforded? 

4.  Has  the  degree  of  protection  which  is  going 
to  be  provided  against  the  effects  of  a 
nuclear  attack  been  determined? 

5.  Have  civil  defense  personnel  made  estimates 
for  physical  casualties  caused  by  atomic, 
biological,  and  chemical  attack,  based  on 
local  data  and  information  furnished  by  the 
Office  of  Civil  Defense? 


6.  Have  civil  defense  personnel  made  estimates  j 

for  psychiatric  casualties  at  the  rate  of  ; 

one  such  casualty  for  every  four  physical 
casualties,  or  less?  6.  ;  / 


7.  Are  plans  for  highway  construction  and  street 

improvements  to  be  coordinated  with  civil  , 

defense  planning  to  enhance  access  to  the  J 

shelters?  7*  ‘  / 

8.  Are  building  codes  such  that  it  is  feasible  ! 

to  include  shelter  space  in  new  buildings?  8.  j  J? 


Figure  4.  Sample  Page  of  Items  from  Plan  Factor  Criticality 
Data  Collection  Form 
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APPENDIX  B:  DATA  ANALYSIS 


IHTOOTWCTION 


This  section  of  the  report  consists  of  a  more  detailed  presen¬ 
tation  uf  the  data  analysis  phase  of  the  study.  The  essential  features 
of  the  analysis  are  summarized  briefly  in  the  section  pertaining  to 
development  of  the  Evaluation  Instrument.  A  discussion  of  the  results 
and  implications  of  the  analysis  is  also  provided  in  that  section. 

Specific  aspects  of  the  data  analysis  covered  here  are:  prelim¬ 
inary  data  processing,  (b)  the  primary  analysis  phase,  and  (c)  deri¬ 
vation  of  item  veights.  The  discussions  regarding  these  efforts  deal 
with  what  was  accomplished,  and,  when  not  obvious,  why  the  approach 
was  used.  Interspersed  throughout  the  discussions  are  figures  illus¬ 
trating  the  analysis  and  results. 


PRELIMINARY  DATA  PROCESSING 


The  item  ratings  in  each  data  collection  form  were  tabulated  by  at 
least  two  project  staff  members.  Several  checks  for  omitted  ratings  or 
incorrect  tallies  were  made  as  the  raw  data  were  prepared  for  analvsis. 

Raw  score  ratings  were  entered  in  a  data  organization  form  where 
they  were  identified  by  rater,  plan  factor  item,  and  category.  A  frequency 
distribution  matrix  of  the  ratings  was  also  prepared  for  each  rater.  Rat¬ 
ings  were  arranged  ty  criticality  level  (0,  1,  2,  and  3)  a»d  category.  The 
frequency  distribution  matrices  made  it  possible  to  compare  rating  patterns 
across  raters  and  identify  those  raters  exhibiting  a  constant  rating  bias. 

Several  instances  of  rater  bias  towards  high  or  low  ratings  were 
identified.  Since  the  bias  appeared  to  be  independent  of  the  'onditions 
under  which  the  ratings  were  assigned;  it  was  decided  to  transform  the  raw 
score  ratings  to  standard  scores.  Utilization  of  standard  scores  would 
equate  for  rater  bias  without  degrading  the  usefulness  of  the  ratings.  The 
raw  score  criticality  levels  (0,  1,  2,  and  3)  were  converted  to  standard 
scores  for  each  rater,  using  as  a  basis  for  the  transformation  the  mean  and 
standard  deviation  of  the  rater's  frequency  distribution. 

The  mean  and  standard  deviation  of  any  standard  score  distribution 
are,  by  definition,  0  and  1,  respectively.  However,  each  rater's  distribu¬ 
tion  of  standard  score  criticality  levels  was  given  a  mean  of  10  to  avoid 
minus  scoret  for  the  lower  criticality  levels. 

Throughout  the  remainder  of  this  repo. t,  all  references  to  ratings, 
scores,  or  data  will  be  to  their  standard  score  form  unless  otherwise  stated. 


DATA  ANALYSIS 


The  criticality  ratings  were  organized  for  computer  analysis  so 
that  all  rating  data  pertinent  to  a  single  plan  factor  item  could  he  placed 
on  one  card.  A  computer  program  already  available  was  modified  slightly  to 
obtain  the  following  information  for  each  factor: 

a)  mean  ratings  and  standard  deviations  for  each  item 
under  conditions  1  and  2. 

b)  a  mean  and  standard  deviation  for  each  item  based  on 
the  combined  ratings  under  both  conditions. 

c)  a  "t"  value  for  each  item  from  the  comparison  of  the 
mean  ratings  under  the  two  conditions. 

Means  and  standard  deviations  were  also  to  be  obtained  for  three 
anticipated  populations  of  ratings.  These  were:  (a)  a  population  of  con¬ 
dition  1  item  means  for  those  items  found  to  be  rated  significantly  differ¬ 
ent,  under  the  two  conditions;  (b)  a  population  of  condition  2  item  means 
for  the  same  items,  and  (c)  a  population  of  combined  mean  ratings  for  those 
items  not  rated  significantly  different. 

The  rationale  for  obtaining  the  above-mentioned  data  was  as  follows. 
Mean  criticality  ratings  were  to  be  used  as  the  basis  for  determining 
weights  for  the  items  in  the  Evaluation  Instrument.  If  the  mean  rating  for 
an  item,  under  condition  1,  was  significantly  higher  than  the  mean  rating 
under  condition  2,  the  two  mean  ratings  would  be  kept  separate  and  two  sets 
of  weights  would  be  derived  for  that  item.  If  the  mean  ratings  were  not 
significantly  different,  all  ratings  would  be  utilized  in  computing  a  com¬ 
bined  mean.  This  value  would  then  be  used  in  deriving  a  single  item  weight. 
The  means  and  standard  deviations  for  the  three  anticipated  populations 
(conditions  1  and  2  means,  and  combined  means)  would  be  used  in  the  deriva¬ 
tion  of  three  different  sets  of  item  weights. 

Computation  of  the  means,  standard  deviations,  and  the  t  values  was 
accomplished  on  a  Burroughs  220  computer. 

A  P  value  of  .01  was  set  as  the  level  of  significance  at  which  the 
null  hypothesis  (no  difference  between  ratings  under  the  two  conditions) 
would  be  rejected.  Since  only  those  instances  in  which  the  mean  rating  for 
condition  1  was  greater  than  that  for  condition  2  were  of  interest,  the 
significant  t  value  for  a  one-tailed  test  was  used. 


An  examination  of  the  t  values  shewed  none  of  the  items  to  have  been 
rated  significantly  different  in  the  appropriate  direction  under  the  two 
widely  differing  conditions.  The  exceptions  to  these  findings  were  seven 
items  where  the  instructions  to  the  raters  made  a  significant  difference 
inevitable.  As  a  double  check,  Fisher's  Exact  Probability  Test  was  applied 
to  the  raw  score  rating  pattern  for  each  item.  Again,  the  seven  items 
previously  mentioned  plus  two  additional  items  were  found  to  be  rated  sig¬ 
nificantly  different  at  the  .01  level.  Analysis  of  the  subject  matter  con¬ 
tent  of  the  two  additional  items  indicated  their  significance  was  a  chance 
occurrence.  This  was  not  surprising  since  at  the  .01  level,  five  (5)  out 
of  a  total  of  493  items  could  be  expected  to  be  significant  by  chance  alone. 
The  three  raw  score  patterns  which  would  be  significant  at  the  .01  level 
(P  value  of  .005  for  a  one-tailed  test)  are  shown  below  in  Figure  5. 
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Figure  5.  Significant  Distributions  of  Raw  Score  Ratings 
Using  the  Fisher  Exact  Probability  Test 
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DERIVATION  OF  ITEM  WEIGHTS 


Since  no  significant  differences  were  found  between  mean  ratings 
for  the  two  conditions,  except  as  noted,  combined  mean  ratings  for  the 
factors  could  be  used  to  derive  a  single  set  of  weights  for  the  plan  factor 
items.  A  frequency  distribution  of  the  combined  mean  ratings  for  the  1+93 
items  is  shown  in  Figure  6.  The  limits  on  the  mean  rating  scale  depict  the 
lowest  possible  mean  standard  score  (8.0)  an  item  could  receive  if  all 
raters  assigned  the  item  a  raw  score  rating  of  0,  and  the  highest  possible 
score  (11.5)  if  all  raters  assigned  the  item  a  raw  score  rating  of  3*  Al¬ 
though  individual  mean  ratings  are  continuous  variables,  they  are  grouped 
in  one-tenth  intervals  for  purposes  of  preparing  the  frequency  distribution 
and  computing  item  weights. 

Before  determining  the  mechanics  for  deriving  item  weights,  the 
following  question  had  to  be  answered.  Should  the  transformation  of  mean 
ratings  to  weights  be  straight-line  or  curvelinear?  A  positively  accel¬ 
erated  curvelinear  transformation  was  chosen  for  the  following  reasons: 

a)  The  scale  used  as  a  basis  for  obtaining  criticality  data 
on  the  plan  factors  (see  Figure  1,  Appendix  A)  exhibited 
a  positively  accelerated  increase  in  level  of  criticality 
from  the  low  to  the  high  end  of  the  scale. 

b)  A  positively  accelerated  curvelinear  transformation  would 
result  in  item  weights  commensurate  with  the  objectives  of 
the  scoring  method. 

In  summary,  the  pertinent  scoring  method  objectives  were: 

— factors  Judged  highly  critical  should  have  a  weight 
reflecting  this  importance, 

--providing  for  less  important  factors  should  not  numer¬ 
ically  equate  for  omitting  a  highly  critical  factor 
whose  absence  cou.’d  lead  to  system  failure,  and 
--heavy  penalties  for  failing  to  provide  for  factors 
whose  omission  could  result  in  system  failure. 

An  approximation  of  the  transformation  chosen  (X  -  8.l)  ,  where  X 
represents  the  mean  standard  score  rating,  is  illustrcted  in  Figure  7* 
Actual  score  transformation  was  accomplished  utilizing  the  conversion  table 
presented  in  Table  1.  Examination  of  Table  1  shows  that  mean  ratings  below 
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Figure  6.  Frequency  Distribution  of  Mean  Standard  Score  Ratings 
of  Plan  Factor  Criticality  Levels 
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Table  1.  Conversion  Table  Used  in  Transforming  Mean  Stand¬ 
ard  Score  Ratings  to  Item  Weights 
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8.5  received  a  weight  of  zero.  This  meant  that  the  presence  or  absence  of 
plan  factors  with  a  mean  criticality  rating  below  8.5  would  have  no  effect 
on  the  operational  capability  of  a  shelter  system.  Six  items  fitting  this 
description  were  deleted  from  the  Evaluation  Instrument.  The  mean  ratings 
of  these  six  items  are  identified  on  the  frequency  distribution  in  Figure  6 
by  the  broken  line  at  the  lower  end  of  the  distribution  separating  them 
from  the  rest  of  the  scores . 

Further  examination  of  Figure  7,  particularly  with  respect  to  the 
higher  mean  ratings,  illustrates  the  effectiveness  cf  the  transformation  in 
achieving  the  desired  relationship  between  ratings  and  weights. 

The  decision  was  macle  to  assign  an  item  weight  of  100  to  those  items 
whose  absence  was  judged  to  result  in  system  failure  by  at  least  92$  of  the 
raters.  When  coupled  with  the  other  aspe  ts  of  the  scoring  method,  such  a 
decision  meant  that  omitting  a  factor  so  weighted  would  be  indicative  of 
system  failure  regardless  of  provisions  for  other  factors  in  its  category. 
If  92$  of  the  raters  gave  an  item  a  raw  score  rating  of  3>  the  minimum 
standard  score  possible  was  11.26.  This  value,  then,  served  as  the  lower 
limit  for  mean  standard  scores  which  could  be  assigned  a  weight  of  100. 
Since  an  item  weight  of  100  was  equivalent  to  denoting  system  failure  if 
the  factor  were  not  provided  for,  mean  ratings  above  11.26  were  restricted 
to  item  weights  of  100.  Mean  ratings  so  weighted  are  set  apart  from  the 
other  scores  plotted  in  Figure  6  by  the  broken  line  at  the  upper  end  of  the 
frequency  distribution. 


APPENDIX  C:  UTILIZATION  OF  THE 

EVALUATION  INSTRUMENT 


UTILIZATION  OF  THE  EVALUATION  INSTRUMENT 


A  detailed  discussion  of  the  intended  use  of  the  Evaluation  Instru¬ 
ment  is  presented  in  this  appendix  as  a  supplement  to  '  ’-e  discourse  on  eval¬ 
uation-instrument  development  and  verification  contained  in  the  main  body 
of  the  report,  This  section  deals  with  (a)  the  purpose  of  conducting  a  com¬ 
munity  shelter  system  evaluation  and  by  whom  it  should  be  conducted,  (b) 
evaluation  techniques,  (c)  scoring  techniques,  (d)  preparation  and  interpre¬ 
tation  of  profiles,  (e)  critiquing  an  evaluation,  and  (f)  updating  techniques. 

Purpose 

As  stated  earlier  in  this  report,  a  major  goal  of  this  project  was  to 
develop  an  evaluation  instrument  which  could  be  used  to  assess  the  opera¬ 
tional  capability  of  community  fallout  shelter  systems.  This  purpose  has 
been  accomplished.  The  Evaluation  Instrument  is  a  fact.  Now  a  more  defini¬ 
tive  purpose  regarding  the  employment  of  the  Evaluation  Instrument  is  in 
order.  Brief  statements  of  portions  of  this  definitive  purpose  have  appeared 
elsewhere  in  this  report.  They  will  be  reviewed  and  further  discussed  here. 

First,  the  Evaluation  Instrument  is  to  be  used  as  a  diagnostic  tool. 

In  this  respect,  it  will  be  useful  in  helping  fulfill  the  federal  responsi¬ 
bility  to  know  and  repoit  the  status  of  civil  defense  efforts  throughout  the 
nation.  Results  of  evaluations  can  be  used  to  improve  shelter  systems  rather 
than  merely  comparing  systems  as  to  their  adequacy.  All  aspects  of  its  devel¬ 
opment  have  been  directed  to  these  ends.  Plan  factors  are  numerous  (487)  and 
to  a  considerable  degree  specific  in  nature.  The  scoring  method  is  based  on 
obtaining  separate  scores  for  each  category  as  opposed  to  a  single  numerical 
score  for  the  entire  shelter  system.  A  profile  chart  was  developed  as  a  means 
of  depicting  the  adequacies  or  deficiencies  of  a  shelter  system  b£  category. 

A  critique  of  each  evaluation  ia  suggested  as  a  means  of  presenting  the  re¬ 
sults  in  a  comprehensive  fora  whi-h  will  be  of  value  to  the  local  civil 
defense  planners  concerned  in  correcting  any  deficiencies  detected. 

Second,  the  Evaluation  Instrument  is  intended  for  use  by  Office  of 
civil  Defense  personnel  at  the  national,  regional  and  state  levels.  It  is 
not  intended  for  use  by  local  civil  defense  officials,  people  Involved  in 
the  development  of  a  shelter  system  for  their  community  cannot  be  expected 
to  possess  the  background  or  th»  ucoeeanvy  to  provide  a  fair  and 


impartial  evaluation  of  their  own  shelter  system.  In  the  future,  it  is 
expected  that  as  shelter  systems  are  developed,  plans  and  descriptions  will 
be  submitted  to  the  Office  of  Civil  Defense  for  evaluation  in  much  the  same 
manner  that  shelter  facility  designs  are  submitted  for  evaluation  and 
approval  at  the  present  time. 

Evaluation  Technique 

As  a  prelude  to  the  actual  evaluation  of  a  shelter  system,  the  fol¬ 
lowing  steps  should  be  taken.  First,  the  evaluator  should  familiarize  him¬ 
self  with  all  material  pertinent  to  the  system  under  evaluation.  Second, 
the  material  should  be  classified  and  organize  d  to  parallel  the  Evaluation 
Instrument.  When  the  second  step  is  no:  possible,  additional  time  should  be 
spent  on  becoming  familiar  with  the  organization  of  the  material,  including 
perhaps  placing  category  labels  in  the  margins  of  the  documents.  Accomplish¬ 
ing  these  two  preliminary  steps  will  greatly  facilitate  the  evaluation  and 
reduce  the  possibility  of  under-evaluating  a  shelter  system  by  overlooking 
information  pertaining  to  some  of  the  plan  factors. 

Item  format  and  its  relation  to  the  evaluation  technique  was  dis¬ 
cussed  previously  in  relation  to  other  aspects  of  Evaluation  Instrument 
format.  The  mechanics  of  shifting  verb  tenses  as  necessary  within  individ¬ 
ual  plan  factor  items  is  covered  in  the  Instruction  Manual.  No  further 
discussion  on  this  point  is  required. 

Determining  whether  or  not  a  specific  "actor  has  been  provided  for 
can  be  either  a  simple  or  a  difficult  task.  Much  depends  on  the  way  infor¬ 
mation  on  the  system  is  classified  and  organized.  Also,  general  rather  than 
specific  descriptions  of  how  individual  factors  are  provided  for  will  affect 
the  stability  of  an  evaluation.  That  is,  if  an  evaluator  is  forced  to  make 
inferences  as  to  whether  or  not  factors  are  provided  for  because  information 
is  stated  in  to?  general  terms,  he  may  not  always  make  the  same  decisions, 
given  like  amounts  of  infonaation  regarding  plan  factors.  The  most  equitable 
solution  to  this  orotlcm  i3  consistency.  Type  and  amount  of  information 
va'.ck  must  be  available  before  a  factor  can  be  considered  as  having  been  pro¬ 
vided  for  should  b?  determined  before  starting  nr.  evaluation.  All  decisions 
•  gar ding  presence  or  absence  of  facto:*;  should  then  be  based  on  these  pre¬ 
determined  guidelines. 

An  Important  point  to  remember  when  evaluating  a  shelter  system  is 
that  deeir. Ions  ifgardlrg  plan  f*«ct>*, •  items  are  t r  1  chet emeus  (present- 


absent-not  applicable)  rather  than  dichotomous  (present -absent) .  Many 
plan  factors  are  applicable  to  all  shelter  systems  regardless  of  the  con- 
ditlone  under  which  the  system  functions,  but  many  are  not.  Several  factors 
may  render  a  particular  item  not  applicable  to  the  system  under  evaluation. 
Among  these  are; 

a)  assumptions  regarding  basic  system  objectives  which  preclude 
the  necessity  of  providing  for  the  factor, 

b)  documented  or  obvious  conditions  such  as  proximity  to  targets, 
extremely  small  shelter  system  in  terms  of  number  of  shelterees 
and  shelters,  etc.,  and 

c)  providing  for  other  factors  in  such  a  manner  that  the  factor 
under  consideration  is  not  applicable  as  stated. 

With  respect  to  (a)  above,  it  must  be  remembered  that  at  least  those 
portions  of  the  community  shelter  system  developed  and  financed  by  the  com¬ 
munity  are  under  its  Jurisdiction.  If  assumptions  are  made  regarding  these 
areas  which  render  plan  factors  not  applicable,  then  as  far  as  that  shelter 
is  concerned  those  factors  are  not  applicable.  If  conditions  are  such  that 
the  factors  should  be  provided  for,  this  point  should  be  emphasized  when 
critiquing  the  evaluation.  But  in  scoring  the  system,  the  factors  in 
question  should  be  marked  "not  applicable." 

One  additional  aspect  of  the  evaluation  technique  should  be  empha¬ 
sized.  This  concerns  control  items  in  the  Evaluation  Instrument.  Several 
of  the  categories  contain  items  which  determine  the  response  to  several  or 
all  items  following  them  in  the  category  when  they  are  responded  to  nega¬ 
tively. 

The  interrelated  nature  of  control  items  and  the  items  they  affect 
is  such  that  a  negative  response  to  a  control  Item  makes  only  a  "negative" 
or  "not  applicable"  response  possible  for  the  affected  items.  For  the 
purpose  of  scoring  the  affected  items,  a  "not  applicable"  response  was  made 
mandatory  when  the  relevant  control  item  received  a  negative  response. 

This  vn»  done  for  two  reasons.  First,  the  affected  items  're  in  actuality 
more  specific  aspects  of  the  control  item  and  are  therefore  in  a  very  real 
sense  no  longer  applicant*  when  the  control  item  is  answered  negatively. 
Second,  there  la  no  practical  value  In  subtracting  the  weights  of  these 


subsidiary  itemo;  to  do  so  would,  in  effect,  inej-easc  the  negative  critical¬ 
ity  score  beyond  the  value  originally  assigned  based  upon  the  Judged  effect 


of  the  control  item. 

Other  aspects  of  the  evaluation  technique,  such  as  the  procedure 
for  marking  individual  items,  are  adequately  covered  in  the  Instruction 
Manual;  they  merit  no  additional  discussion  here. 

Scoring  Technique 

Detailed  procedures  for  scoring  an  evaluation  are  presented  in  the 
Instruction  Manual.  This  discussion  is  intended  to  be  expletive  in  nature 
and  useful  as  a  supplement  to  the  procedures  outlined. 

As  previously  explained,  there  are  three  response  choices  for  each 
plan  factor  item  in  the  Evaluation  Instrument.  They  are:  "not  applicable, " 
'yes,"  and  "no."  The  symbols  used  to  indicate  these  responses  in  the 
instrument  are  N/A,  Y,  and  a  number  representing  the  item  weight.  Item 
weights  are  used  as  symbols  for  a  "no"  response  for  two  reasons.  First, 
the  negative  response  is  the  only  one  given  a  numerical  value  in  the  scoring 
method.  Second,  it  simplifies  the  format  of  the  Evaluation  Instrument  and 
makes  unnecessary  a  further  translation  of  a  "no"  response  to  a  numerical 
value  when  scoring  an  evaluation.  With  this  format,  it  is  a  3imple  two- 
step  procedure  to  obtain  a  converted  score  for  each  category.  All  item 
weights  in  a  category  which  are  circled  during  the  evaluation  (in'’1' eating 
a  negative  response)  are  added  together.  This  sum  is  then  subtracted  from 
the  maximum  score  for  the  category  to  obtain  the  converted  score. 

Coincident  with  the  negatively  oriented  scoring  procedure  is  the 
negative  nature  of  the  item  weights  themselves.  That  is,  they  are  based  on 
judgments  regarding  the  effect  of  the  failure  to  provide  (or  to  provide 
acceptably)  for  plan  factors  in  a  shelter  system.  Also,  the  weights  are 
subtracted  from  a  maximum  possible  score  vhen  the  corresponding  plan  factors 
are  not  provided  for  in  the  shelter  system  being  evaluated.  The  rationale 
for  this  negative  approach  to  system  evaluation  is  based  on  the  scoring 
method  objectives  previously  outlined.  Prinr'paljy,  it  is  the  most  effec¬ 
tive  way  of  high-lighting  shelter  system  def  icienct»»s . 

As  previously  mentioned,  ar.  arbitrary  value  of  100  was  assigned  to 
vveh  categ  ory  as  the  maximum  possible  score  a  category  could  achieve.  A 
■utegory  receives  the  maximum  ocore  only  if  all  factors  in  the  category  are 
provided  for,  or  those  which  are  not  provided  Tor  are  not  applicable  to  the 
shelter  system  being  evaluated.  For  every  applicable  Plan  factor  nut  pro- 


vided  for  in  a  category,  an  amount  equal  to  the  factor's  item  weight  is  sub¬ 
tracted  from  100, 

Since  the  sum  of  all  item  weights  in  a  category  is  in  most  cases 
greater  than  100,  it  is  quite  possible  for  a  category  to  have  a  negative 
converted  score.  A  aero  or  negative  converted  score  for  a  category  can 
occur  when  one  or  more  factors  judged  unanimously  as  causing  system  failure 
are  not  provided  for,  or  in  the  case  of  the  larger  categories  when  several 
moderately  critical  plan  factors  are  not  provided  for. 

In  summary,  the  scoring  technique  can  be  likened  to  a  demerit  system 
in  which  each  person  starts  with  a  given  number  of  credits  or  merits.  Then 
a  pre -determined  number  of  points  are  assigned  to  various  negative  character 
istics,  and  demerits  of  corresponding  value  are  charged  to  individuals  for 
deficiencies  such  as  slovenly  appearance,  below  average  performance,  etc. 

In  such  a  system,  the  best  record  which  can  be  achieved  is  to  end  up  with 
the  same  number  of  points  originally  assigned.  Anything  less  denotes  the 
detection  of  deficiencies..  So  it  is  with  the  scoring  method  developed  for 
the  Evaluation  Instrument. 

Profile  Chart 


Upon  completion  of  the  computational  steps  in  scoring  an  evaluation, 
a  converted  score  should  appear  on  the  Computation  Sheet  of  the  Evaluation 
Instrument  booklet  for  each  category.  This  converted  score  is  plotted  on 
the  profile  chart  provided  in  the  Evaluation  Instrument  booklet.  An  example 
of  a  completed  profile  chart  is  shown  in  Fig'.jre  8. 

Because  item  weights  are  based  upon  pooled  judgments  rather  than 
upon  experimental  manipulation  of  operational  shelter  characteristics,  no 
system  failure  point  has  been  labelled  on  the  profile  chart  scale.  However, 
the  item  weights,  maximum  score  concept,  and  profile  chart  scale,  have  been 
developed  and  defined  in  such  a  way  that  a  zero  or  negative  converted  score 
in  one  or  more  categories  is  indicative  of  system  failure,  Since  the  Evalua¬ 
tion  Instrument  is  a  diagnostic  tool,  primary  concern  is  for  whet  caused  an 
indication  of  system  failure.  Categories  deficient  to  the  point  of  system 
failure  are  quite  obviously  portrayed  as  such  on  the  profile  chart.  This 
Js  the  function  of  the  present  profile  chart.  Identification  and  discussion 
of  the  causal  factors  and  the  suggestion  of  remedial  action  are  the  function 
of  the  Evaluation  Critique. 


Evaluation  Critique 


It  is  not  the  purpose  of  this  section  to  provide  a  detailed  outline 
of  how  an  evaluation  should  he  critiqued.  The  special  re.l ationship  of  the 
Office  of  Civil  Defense  (OCD)  with  state  and  local  civil  defense  organiza¬ 
tions  demands  that  this  be  determined  solely  by  OCD  policy.  However, 
several  suggestions  are  discussed  regarding  some  aspects  Judged  to  be 
important  in  such  a  critique  as  it  relates  to  the  evaluation. 

The  most  important  point,  quite  naturally,  i3  that  a  critique  be 
prepared.  No  score  in  a  diagnostic  evaluation  such  as  this  has  meaning 
unless  the  details  of  the  shelter  system  to  which  the  score  relates  are 
discussed. 

Each  category  score  should  be  discussed  in  terms  of  the  plan  factors 
which  are  absent  or  inadequately  provided  for.  For  the  critique  to  be  of 
the  most  value,  it  is  suggested  that  detailed  guidance  also  be  given  on 
how  missing  factors  can  be  provided  for.  If  alternative  means  of  provid¬ 
ing  for  factors  are  available,  they  should  be  listed  so  that  communities 
can  choose  the  means  most  suitable  to  the^r  circumstances. 

When  a  category  receives  a  zero  or  negative  score,  one  or  more  highly 
critical  factors  are  usually  missing,  and  the  remedial  action  required  is 
obvious  and  amenable  to  discussion.  However,  interpretation  of  low  positive 
scores  with  respect  to  system  failure  is  a  much  more  complicated  task. 

Since  certain  combinations  of  absent  factors  could  result  in  equal  scores 
but  unequal  total  effects,  inspection  of  the  individual  factors  involved 
should  always  be  accomplished.  If  a  low  positive  score  is  the  result  of 
failing  to  provide  for  a  considerable  number  of  moderate  and  low  criticality 
factors,  suggestions  for  improving  the  system  may  have  to  be  presented  in 
the  form  of  alternative  courses  of  action,  particularly  when  factors  which 
are  low  in  criticality  but  costly  to  provide  for  are  involved. 

Updating  Techniques 

The  content  of  plan  factors  included  in  the  present  Evaluation  Instru¬ 
ment  reflects  material  in  OCD  issuances,  and  in  CD  research  reports  available 
as  of  the  end  of  November  1962.  Inputs  to  the  factors  themselves  were  curtailed 
at  this  tine  to  permit  the  collection  of  data  on  the  level  of  criticality 
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of  the  individual  factors.  Civil  defense  research  as  an  on-going  effort, 
has  already  produced  results  which  affect  the  content  of  the  present  plan 
factors. 

The  extent  to  which  the  present  population  of  plan  factors  represents 
the  total  possible  number,  is  not  known.  However,  it  is  known  that  coverage 
is  not  complete;  particularly  in  what  might  be  termed  the  "extra -shelter" 
areas.  To  further  complicate  the  matter  of  coverage,  no  two  civil  defense 
"experts"  will  agree  on  all  factors  which  should  or  should  not  be  included 
in  an  evaluation  instrument;  seldom  will  they  agree  on  the  importance  of  a 
given  factor. 

In  light  of  the  above-mentioned  conditions,  the  revision  of  item 
content,  and  the  addition  or  deletion  of  whole  items  must  be  feasible  and 
j.  datively  easy  in  order  to  maintain  a  current  and  acceptable  instrument. 
Three  methods  of  determining  item  weights  for  new  or  substantially  revised 
items  are  offered  in  this  section.  The  weighting  techniques  described  herein 
represent  interim  measures  which  are  appropriate  for  a  limited  number  of 
changes.  If  large-scale  revisions  are  to  be  undertaken,  efforts  should  in¬ 
clude  the  development  of  a  more  sophisticated  scoiing  method.  Some  aspects 
pertinent  to  refining  the  present  scoring  method  are  outlined  in  the  section 
of  this  report  dealing  with  suggestions  for  future  research. 

Approach  A.  The  approach  outlined  here  is  essentially  that  used  to 
derive  item  weights  for  the  present  evaluation  instrument.  It  is  recom¬ 
mended  as  the  most  ...liable  of  the  three  to  be  presented.  The  sequential 
steps  in  implementing  the  approach  are  outlined  below. 

1.  Obtain  ratings  regarding  the  level  of  criticality  of  each 
new  or  substantially  revised  item  from  as  many  Judges  as 
possible  who  are  knowledgeable  in  the  subject-matter  areas 
represented.  The  raters  should  be  given  instructions  similar 
to  those  shown  in  Figure  1,  Appendix  A.  The  i terns  need  not 
be  rated  under  specific  conditions  as  was  done  in  the  present 
study  since  it  was  found  that  such  conditions  do  not  sig¬ 
nificantly  affect  the  ratings. 

2.  Compute  standard  score  equivalents  for  each  rater  for  the 
four  levels  of  critical ity  (0,  1,  2,  and  3)* 
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3.  Sum  the  standard  score  ratings  across  raters  for  oach  item. 
Calculate  a  mean  standard  score  rating  for  each  item.  Round 
these  mean  ratings  to  the  nearest  tenth  (examples:  8.^,  9*I> 

10.6,  etc.) 

4.  Calculate  a  conversion  table  similar  to  the  one  shown  in 
Table  1,  Appendix  B,  using  the  transformation  (X-K)S  X 
represents  the  mean  standard  score  rating  and  K  if-  a  constant. 

The  value  of  the  constant  should  equal  the  mean  standard  score 
rating  obtained  if  all  raters  assigned  an  item  a  raw  score 
rating  of  0. 

5.  Enter  the  conversion  table  with  the  value  of  U»@  meau  stand¬ 
ard  score  rating  for  the  item,  and  read  off  the  correap'-n-Hng 
item  weight.  Using  the  transformation  suggested  in  step  4, 
an  item  which  receives  a  majority  of  raw  score  ratings  of  0 
will  have  an  item  weight  of  0.  This  signifies  that  the  item 
is  Judged  to  have  no  effect  cn  the  operational  capability 

of  a  system.  It  should  not,  therefore,  be  included  in  the 
evaluation  instrument.  , 

;i 

Approach  B.  The  approach  outlined  here  is  a  short-cut  method  for 
obtaining  item  weights  similar  to  Approach  A.  Its  use,  however,  demands 
acceptance  of  the  assumption  that,  statistically  speaking,  the  raters  to  be 
used  to  assess  the  criticality  of  new  items  sure  from  the  same  rater  popu¬ 
lation  as  those  used  in  Judging  item  criticality  for  the  present  evaluation 
instrument.  Accepting  this  assumption  makes  unnecessary  the  (a)  computation 
of  a  new  set  of  standard  scores  for  each  rater,  and  (b)  development  of  a  new 
conversion  table.  The  steps  to  be  taken  in  implementing  Approach  B  are  as 
follows . 

1.  Obtain  raw  score  ratings  of  item  criticality  as  described 
in  Approach  A. 

2.  Substitute  the  standard  score  equivalents  given  below  for 
each  rater's  raw  aeo»v  ratings. 

Rav  Score  Levels  Standard  Score  Equivalents 

Q<0 


0 


3.  Sum  standard  score  equivalents  across  raters  for  each  item. 
Compute  a  mean  standard  score  rating  for  each  item.  Round  the 
mean  standard  score  ratings  to  the  nearest  tenth. 

4.  Enter  the  conversion  table  shown  in  Table  1,  Appendix  B,  with 
the  mean  standard  score  rating,  and  read  off  the  corresponding 
item  weight. 

Approach  C.  If  the  two  preceding  approaches  are  not  practicable, 
a  third  method  may  be  used.  However,  it  is  the  least  desirable  of  the  three 
from  the  standpoint  of  reliability  of  results.  The  steps  for  this  approach 
are  outlined  below. 

1.  Compare  each  new  item  with  other  plan  factor  items  of  the 
same  subject-matter  background. 

2.  If  a  comparable  item  (in  terms  of  Judged  importance)  can  be 
found  in  the  existing  instrument,  assign  the  new  item  the 
same  weight. 

3.  If  a  comparable  item  can  not  be  found,  pick  out  two  items; 
one  which  appears  to  be  slightly  more  important  tnan  the 
item  under  consideration,  and  the  other  slightly  less  im¬ 
portant.  Assign  the  new  item  a  weight  mid-way  between 
the  weights  of  the  two  comparison  items. 

General  Comments.  Minor  changes  in  item  content  should  not  affect 
item  criticality.  An  example  of  such  a  change  would  be  the  reduction  in 
the  protection  factor  minimum  acceptable  in  the  federal  marking  and  stocking 
program  from  100  to  40.  However,  when  standards  or  plan  factors  are  altered 
substantially,  the  affected  factors  should  be  re-evaluated  in  terms  of  their 
relative  importance  to  shelter  system  operation.  It  should  also  be  re¬ 
emphasized  that  the  above-mentioned  approaches  to  obtaining  weights  for  new 
items  to  be  added  to  the  present  evaluation  instrument  are  acceptable  when 
small  numbers  of  items  are  involved.  Wholesale  changes  in  the  content  of 
the  instrument  should  be  accompanied  by  repeating  the  total  weighting 
procedures  used  in  the  present  study;  such  an  effort  could  also  include 
refining  the  present  scoring  method. 
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